Date: (Mon) May 30, 2016

Introduction:

Data: Source: Training: “https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/train2016.csv”
New: “https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/test2016.csv”
Time period:

Synopsis:

Based on analysis utilizing <> techniques, :

Summary of key steps & error improvement stats:

Prediction Accuracy Enhancement Options:

transform.data chunk:
- derive features from multiple features
manage.missing.data chunk:
- Not fill missing vars
- Fill missing numerics with a different algorithm
- Fill missing chars with data based on clusters

[](.png)

Potential next steps include:

Organization:
- Categorize by chunk
- Priority criteria:
  1. Ease of change
  2. Impacts report
  3. Cleans innards
  4. Bug report
all chunks:
- at chunk-end rm(!glb_)
manage.missing.data chunk:
- cleaner way to manage re-splitting of training vs. new entity
extract.features chunk:
- Add n-grams for glbFeatsText
  - “RTextTools”, “tau”, “RWeka”, and “textcat” packages
fit.models chunk:
- Classification: Plot AUC Curves for all models & highlight glbMdlSel
- Prediction accuracy scatter graph:
- Add tiles (raw vs. PCA)
- Use shiny for drop-down of “important” features
- Use plot.ly for interactive plots ?
- Change .fit suffix of model metrics to .mdl if it’s data independent (e.g. AIC, Adj.R.Squared - is it truly data independent ?, etc.)
- create a custom model for rpart that has minbucket as a tuning parameter
- varImp for randomForest crashes in caret version:6.0.41 -> submit bug report
Probability handling for multinomials vs. desired binomial outcome
ROCR currently supports only evaluation of binary classification tasks (version 1.0.7)
extensions toward multiclass classification are scheduled for the next release
fit.all.training chunk:
- myplot_prediction_classification: displays ‘x’ instead of ‘+’ when there are no prediction errors
Compare glb_sel_mdl vs. glb_fin_mdl:
- varImp
- Prediction differences (shd be minimal ?)
Move glb_analytics_diag_plots to mydsutils.R: (+) Easier to debug (-) Too many glb vars used
Add print(ggplot.petrinet(glb_analytics_pn) + coord_flip()) at the end of every major chunk
Parameterize glb_analytics_pn
Move glb_impute_missing_data to mydsutils.R: (-) Too many glb vars used; glb_<>_df reassigned
Do non-glm methods handle interaction terms ?
f-score computation for classifiers should be summation across outcomes (not just the desired one ?)
Add accuracy computation to glb_dmy_mdl in predict.data.new chunk
Why does splitting fit.data.training.all chunk into separate chunks add an overhead of ~30 secs ? It’s not rbind b/c other chunks have lower elapsed time. Is it the number of plots ?
Incorporate code chunks in print_sessionInfo
Test against
- projects in github.com/bdanalytics
- lectures in jhu-datascience track

Analysis:

rm(list = ls())
set.seed(12345)
options(stringsAsFactors = FALSE)
source("~/Dropbox/datascience/R/mycaret.R")
source("~/Dropbox/datascience/R/mydsutils.R")

## Loading required package: caret

## Loading required package: lattice

## Loading required package: ggplot2

source("~/Dropbox/datascience/R/mypetrinet.R")
source("~/Dropbox/datascience/R/myplclust.R")
source("~/Dropbox/datascience/R/myplot.R")
source("~/Dropbox/datascience/R/myscript.R")
source("~/Dropbox/datascience/R/mytm.R")
# Gather all package requirements here
suppressPackageStartupMessages(require(doMC))
glbCores <- 6 # of cores on machine - 2
registerDoMC(glbCores) 

suppressPackageStartupMessages(require(caret))
require(plyr)

## Loading required package: plyr

require(dplyr)

## Loading required package: dplyr

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:plyr':
## 
##     arrange, count, desc, failwith, id, mutate, rename, summarise,
##     summarize

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

require(knitr)

## Loading required package: knitr

require(stringr)

## Loading required package: stringr

#source("dbgcaret.R")
#packageVersion("snow")
#require(sos); findFn("cosine", maxPages=2, sortby="MaxScore")

# Analysis control global variables
# Inputs
#   url/name = "<PathPointer>"; if url specifies a zip file, name = "<filename>"; 
#               or named collection of <PathPointer>s
#   sep = choose from c(NULL, "\t")
glbObsTrnFile <- list(url = "https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/train2016.csv"
    # or list(url = c(NULL, <.inp1> = "<path1>", <.inp2> = "<path2>"))
    #, splitSpecs = list(method = "copy" # default when glbObsNewFile is NULL
    #                       select from c("copy", NULL ???, "condition", "sample", )
    #                      ,nRatio = 0.3 # > 0 && < 1 if method == "sample" 
    #                      ,seed = 123 # any integer or glbObsTrnPartitionSeed if method == "sample" 
    #                      ,condition = # or 'is.na(<var>)'; '<var> <condition_operator> <value>'    
    #                      )
    )                   
 
glbObsNewFile <- list(url = "https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/test2016.csv") 

glbObsDropCondition <- NULL # : default
#   enclose in single-quotes b/c condition might include double qoutes
#       use | & ; NOT || &&    
#   '<condition>' 
    # 'grepl("^First Draft Video:", glbObsAll$Headline)'
    # 'is.na(glbObsAll[, glb_rsp_var_raw])'
    # '(is.na(glbObsAll[, glb_rsp_var_raw]) & grepl("Train", glbObsAll[, glbFeatsId]))'
    # 'is.na(strptime(glbObsAll[, "Date"], glbFeatsDateTime[["Date"]]["format"], tz = glbFeatsDateTime[["Date"]]["timezone"]))'
#nrow(do.call("subset",list(glbObsAll, parse(text=paste0("!(", glbObsDropCondition, ")")))))
    
glb_obs_repartition_train_condition <- NULL # : default
#    "<condition>" 

glb_max_fitobs <- NULL # or any integer
glbObsTrnPartitionSeed <- 123 # or any integer
                         
glb_is_regression <- FALSE; glb_is_classification <- !glb_is_regression; 
    glb_is_binomial <- TRUE # or TRUE or FALSE

glb_rsp_var_raw <- "Party"

# for classification, the response variable has to be a factor
glb_rsp_var <- "Party.fctr"

# if the response factor is based on numbers/logicals e.g (0/1 OR TRUE/FALSE vs. "A"/"B"), 
#   or contains spaces (e.g. "Not in Labor Force")
#   caret predict(..., type="prob") crashes
glb_map_rsp_raw_to_var <- #NULL 
function(raw) {
#     return(raw ^ 0.5)
#     return(log(raw))
#     return(log(1 + raw))
#     return(log10(raw)) 
#     return(exp(-raw / 2))
    ret_vals <- rep_len(NA, length(raw)); ret_vals[!is.na(raw)] <- ifelse(raw[!is.na(raw)] == "Republican", "R", "D"); return(relevel(as.factor(ret_vals), ref = "R"))
#     as.factor(paste0("B", raw))
#     as.factor(gsub(" ", "\\.", raw))
    }

#if glb_rsp_var_raw is numeric:
#print(summary(glbObsAll[, glb_rsp_var_raw]))
#glb_map_rsp_raw_to_var(tst <- c(NA, as.numeric(summary(glbObsAll[, glb_rsp_var_raw])))) 

#if glb_rsp_var_raw is character:
#print(table(glbObsAll[, glb_rsp_var_raw], useNA = "ifany"))
# print(table(glb_map_rsp_raw_to_var(tst <- glbObsAll[, glb_rsp_var_raw]), useNA = "ifany"))

glb_map_rsp_var_to_raw <- #NULL 
function(var) {
#     return(var ^ 2.0)
#     return(exp(var))
#     return(10 ^ var) 
#     return(-log(var) * 2)
#     as.numeric(var)
#     levels(var)[as.numeric(var)]
    sapply(levels(var)[as.numeric(var)], function(elm) 
        if (is.na(elm)) return(elm) else
        if (elm == 'R') return("Republican") else
        if (elm == 'D') return("Democrat") else
        stop("glb_map_rsp_var_to_raw: unexpected value: ", elm)
        )  
#     gsub("\\.", " ", levels(var)[as.numeric(var)])
#     c("<=50K", " >50K")[as.numeric(var)]
#     c(FALSE, TRUE)[as.numeric(var)]
}
# print(table(glb_map_rsp_var_to_raw(glb_map_rsp_raw_to_var(tst)), useNA = "ifany"))

if ((glb_rsp_var != glb_rsp_var_raw) && is.null(glb_map_rsp_raw_to_var))
    stop("glb_map_rsp_raw_to_var function expected")

# List info gathered for various columns
# <col_name>:   <description>; <notes>
# USER_ID - an anonymous id unique to a given user
# YOB - the year of birth of the user
# Gender - the gender of the user, either Male or Female
# Income - the household income of the user. Either not provided, or one of "under $25,000", "$25,001 - $50,000", "$50,000 - $74,999", "$75,000 - $100,000", "$100,001 - $150,000", or "over $150,000".
# HouseholdStatus - the household status of the user. Either not provided, or one of "Domestic Partners (no kids)", "Domestic Partners (w/kids)", "Married (no kids)", "Married (w/kids)", "Single (no kids)", or "Single (w/kids)".
# EducationalLevel - the education level of the user. Either not provided, or one of "Current K-12", "High School Diploma", "Current Undergraduate", "Associate's Degree", "Bachelor's Degree", "Master's Degree", or "Doctoral Degree".
# Party - the political party for whom the user intends to vote for. Either "Democrat" or "Republican
# Q124742, Q124122, . . . , Q96024 - 101 different questions that the users were asked on Show of Hands. If the user didn't answer the question, there is a blank. For information about the question text and possible answers, see the file Questions.pdf.

# currently does not handle more than 1 column; consider concatenating multiple columns
# If glbFeatsId == NULL, ".rownames <- as.numeric(row.names())" is the default
glbFeatsId <- "USER_ID" # choose from c(NULL : default, "<id_feat>") 
glbFeatsCategory <- "Hhold.fctr" # choose from c(NULL : default, "<category_feat>")
# glbFeatsCategory <- "Q109244.fctr" # choose from c(NULL : default, "<category_feat>") -> OOB performed worse than "Hhold.fctr"

# User-specified exclusions
glbFeatsExclude <- c(NULL
#   Feats that shd be excluded due to known causation by prediction variable
# , "<feat1", "<feat2>"
#   Feats that are factors with unique values (as % of nObs) > 49 (empirically derived)
#   Feats that are linear combinations (alias in glm)
#   Feature-engineering phase -> start by excluding all features except id & category & 
#       work each one in
    , "USER_ID", "YOB", "Gender", "Income", "HouseholdStatus", "EducationLevel" 
    ,"Q124742","Q124122" 
    ,"Q123621","Q123464"
    ,"Q122771","Q122770","Q122769","Q122120"
    ,"Q121700","Q121699","Q121011"
    ,"Q120978","Q120650","Q120472","Q120379","Q120194","Q120014","Q120012" 
    ,"Q119851","Q119650","Q119334"
    ,"Q118892","Q118237","Q118233","Q118232","Q118117"
    ,"Q117193","Q117186"
    ,"Q116797","Q116881","Q116953","Q116601","Q116441","Q116448","Q116197"
    ,"Q115602","Q115777","Q115610","Q115611","Q115899","Q115390","Q115195"
    ,"Q114961","Q114748","Q114517","Q114386","Q114152"
    ,"Q113992","Q113583","Q113584","Q113181"
    ,"Q112478","Q112512","Q112270"
    ,"Q111848","Q111580","Q111220"
    ,"Q110740"
    ,"Q109367","Q109244"
    ,"Q108950","Q108855","Q108617","Q108856","Q108754","Q108342","Q108343"
    ,"Q107869","Q107491"
    ,"Q106993","Q106997","Q106272","Q106388","Q106389","Q106042"
    ,"Q105840","Q105655"
    ,"Q104996"
    ,"Q103293"
    ,"Q102906","Q102674","Q102687","Q102289","Q102089"
    ,"Q101162","Q101163","Q101596"
    ,"Q100689","Q100680","Q100562","Q100010"
    ,"Q99982"
    ,"Q99716"
    ,"Q99581"
    ,"Q99480"
    ,"Q98869"
    ,"Q98578"
    ,"Q98197"
    ,"Q98059","Q98078"
    ,"Q96024" # Done
    ,".pos") 
if (glb_rsp_var_raw != glb_rsp_var)
    glbFeatsExclude <- union(glbFeatsExclude, glb_rsp_var_raw)                    

glbFeatsInteractionOnly <- list()
#glbFeatsInteractionOnly[["<child_feat>"]] <- "<parent_feat>"

glbFeatsDrop <- c(NULL
                # , "<feat1>", "<feat2>"
                )

glb_map_vars <- NULL # or c("<var1>", "<var2>")
glb_map_urls <- list();
# glb_map_urls[["<var1>"]] <- "<var1.url>"

# Derived features; Use this mechanism to cleanse data ??? Cons: Data duplication ???
glbFeatsDerive <- list();

# glbFeatsDerive[["<feat.my.sfx>"]] <- list(
#     mapfn = function(<arg1>, <arg2>) { return(function(<arg1>, <arg2>)) } 
#   , args = c("<arg1>", "<arg2>"))
#myprint_df(data.frame(ImageId = mapfn(glbObsAll$.src, glbObsAll$.pos)))
#data.frame(ImageId = mapfn(glbObsAll$.src, glbObsAll$.pos))[7045:7055, ]

    # character
#     mapfn = function(Education) { raw <- Education; raw[is.na(raw)] <- "NA.my"; return(as.factor(raw)) } 
#     mapfn = function(Week) { return(substr(Week, 1, 10)) }
#     mapfn = function(Name) { return(sapply(Name, function(thsName) 
#                                             str_sub(unlist(str_split(thsName, ","))[1], 1, 1))) } 

#     mapfn = function(descriptor) { return(plyr::revalue(descriptor, c(
#         "ABANDONED BUILDING"  = "OTHER",
#         "**"                  = "**"
#                                           ))) }

#     mapfn = function(description) { mod_raw <- description;
    # This is here because it does not work if it's in txt_map_filename
#         mod_raw <- gsub(paste0(c("\n", "\211", "\235", "\317", "\333"), collapse = "|"), " ", mod_raw)
    # Don't parse for "." because of ".com"; use customized gsub for that text
#         mod_raw <- gsub("(\\w)(!|\\*|,|-|/)(\\w)", "\\1\\2 \\3", mod_raw);
    # Some state acrnoyms need context for separation e.g. 
    #   LA/L.A. could either be "Louisiana" or "LosAngeles"
        # modRaw <- gsub("\\bL\\.A\\.( |,|')", "LosAngeles\\1", modRaw);
    #   OK/O.K. could either be "Oklahoma" or "Okay"
#         modRaw <- gsub("\\bACA OK\\b", "ACA OKay", modRaw); 
#         modRaw <- gsub("\\bNow O\\.K\\.\\b", "Now OKay", modRaw);        
    #   PR/P.R. could either be "PuertoRico" or "Public Relations"        
        # modRaw <- gsub("\\bP\\.R\\. Campaign", "PublicRelations Campaign", modRaw);        
    #   VA/V.A. could either be "Virginia" or "VeteransAdministration"        
        # modRaw <- gsub("\\bthe V\\.A\\.\\:", "the VeteranAffairs:", modRaw);
    #   
    # Custom mods

#         return(mod_raw) }

    # numeric
# Create feature based on record position/id in data   
glbFeatsDerive[[".pos"]] <- list(
    mapfn = function(raw1) { return(1:length(raw1)) }
    , args = c(".rnorm"))
# glbFeatsDerive[[".pos.y"]] <- list(
#     mapfn = function(raw1) { return(1:length(raw1)) }       
#     , args = c(".rnorm"))    

# Add logs of numerics that are not distributed normally
#   Derive & keep multiple transformations of the same feature, if normality is hard to achieve with just one transformation
#   Right skew: logp1; sqrt; ^ 1/3; logp1(logp1); log10; exp(-<feat>/constant)
# glbFeatsDerive[["WordCount.log1p"]] <- list(
#     mapfn = function(WordCount) { return(log1p(WordCount)) } 
#   , args = c("WordCount"))
# glbFeatsDerive[["WordCount.root2"]] <- list(
#     mapfn = function(WordCount) { return(WordCount ^ (1/2)) } 
#   , args = c("WordCount"))
# glbFeatsDerive[["WordCount.nexp"]] <- list(
#     mapfn = function(WordCount) { return(exp(-WordCount)) } 
#   , args = c("WordCount"))
#print(summary(glbObsAll$WordCount))
#print(summary(mapfn(glbObsAll$WordCount)))
    
# If imputation shd be skipped for this feature
# glbFeatsDerive[["District.fctr"]] <- list(
#     mapfn = function(District) {
#         raw <- District;
#         ret_vals <- rep_len("NA", length(raw)); 
#         ret_vals[!is.na(raw)] <- sapply(raw[!is.na(raw)], function(elm) 
#                                         ifelse(elm < 10, "1-9", 
#                                         ifelse(elm < 20, "10-19", "20+")));
#         return(relevel(as.factor(ret_vals), ref = "NA"))
#     }       
#     , args = c("District"))    

# YOB options:
# 1. Missing data:
# 1.1   0 -> Does not improve baseline
# 1.2   Cut factors & "NA" is a level
# 2. Data corrections: < 1928 & > 2000
# 3. Scale YOB
# 4. Add Age
glbFeatsDerive[["YOB.Age.fctr"]] <- list(
    mapfn = function(raw1) {
        raw <- 2016 - raw1 
        # raw[!is.na(raw) & raw >= 2010] <- NA
        raw[!is.na(raw) & (raw <= 15)] <- NA
        raw[!is.na(raw) & (raw >= 90)] <- NA        
        retVal <- rep_len("NA", length(raw))
        # breaks = c(1879, seq(1949, 1989, 10), 2049)
        # cutVal <- cut(raw[!is.na(raw)], breaks = breaks, 
        #               labels = as.character(breaks + 1)[1:(length(breaks) - 1)])
        cutVal <- cut(raw[!is.na(raw)], breaks = c(15, 20, 25, 30, 35, 40, 50, 65, 90))
        retVal[!is.na(raw)] <- levels(cutVal)[cutVal]
        return(factor(retVal, levels = c("NA"
                ,"(15,20]","(20,25]","(25,30]","(30,35]","(35,40]","(40,50]","(50,65]","(65,90]"),
                        ordered = TRUE))
    }
    , args = c("YOB"))

glbFeatsDerive[["Gender.fctr"]] <- list(
    mapfn = function(raw1) {
        raw <- raw1
        raw[raw %in% ""] <- "N"
        raw <- gsub("Male"  , "M", raw, fixed = TRUE)
        raw <- gsub("Female", "F", raw, fixed = TRUE)        
        return(relevel(as.factor(raw), ref = "N"))
    }
    , args = c("Gender"))

glbFeatsDerive[["Income.fctr"]] <- list(
    mapfn = function(raw1) { raw <- raw1;
        raw[raw %in% ""] <- "N"
        raw <- gsub("under $25,000"      , "<25K"    , raw, fixed = TRUE)
        raw <- gsub("$25,001 - $50,000"  , "25-50K"  , raw, fixed = TRUE)
        raw <- gsub("$50,000 - $74,999"  , "50-75K"  , raw, fixed = TRUE)
        raw <- gsub("$75,000 - $100,000" , "75-100K" , raw, fixed = TRUE)        
        raw <- gsub("$100,001 - $150,000", "100-150K", raw, fixed = TRUE)
        raw <- gsub("over $150,000"      , ">150K"   , raw, fixed = TRUE)        
        return(factor(raw, levels = c("N","<25K","25-50K","50-75K","75-100K","100-150K",">150K"),
                      ordered = TRUE))
    }
    , args = c("Income"))

glbFeatsDerive[["Hhold.fctr"]] <- list(
    mapfn = function(raw1) { raw <- raw1;
        raw[raw %in% ""] <- "N"
        raw <- gsub("Domestic Partners (no kids)", "PKn", raw, fixed = TRUE)
        raw <- gsub("Domestic Partners (w/kids)" , "PKy", raw, fixed = TRUE)        
        raw <- gsub("Married (no kids)"          , "MKn", raw, fixed = TRUE)
        raw <- gsub("Married (w/kids)"           , "MKy", raw, fixed = TRUE)        
        raw <- gsub("Single (no kids)"           , "SKn", raw, fixed = TRUE)
        raw <- gsub("Single (w/kids)"            , "SKy", raw, fixed = TRUE)        
        return(relevel(as.factor(raw), ref = "N"))
    }
    , args = c("HouseholdStatus"))

glbFeatsDerive[["Edn.fctr"]] <- list(
    mapfn = function(raw1) { raw <- raw1;
        raw[raw %in% ""] <- "N"
        raw <- gsub("Current K-12"         , "K12", raw, fixed = TRUE)
        raw <- gsub("High School Diploma"  , "HSD", raw, fixed = TRUE)        
        raw <- gsub("Current Undergraduate", "CCg", raw, fixed = TRUE)
        raw <- gsub("Associate's Degree"   , "Ast", raw, fixed = TRUE)
        raw <- gsub("Bachelor's Degree"    , "Bcr", raw, fixed = TRUE)        
        raw <- gsub("Master's Degree"      , "Msr", raw, fixed = TRUE)
        raw <- gsub("Doctoral Degree"      , "PhD", raw, fixed = TRUE)        
        return(factor(raw, levels = c("N","K12","HSD","CCg","Ast","Bcr","Msr","PhD"),
                      ordered = TRUE))
    }
    , args = c("EducationLevel"))

# for (qsn in c("Q124742","Q124122"))
# for (qsn in grep("Q12(.{4})(?!\\.fctr)", names(glbObsTrn), value = TRUE, perl = TRUE))
for (qsn in grep("Q", glbFeatsExclude, fixed = TRUE, value = TRUE))    
    glbFeatsDerive[[paste0(qsn, ".fctr")]] <- list(
        mapfn = function(raw1) {
            raw1[raw1 %in% ""] <- "NA"
            rawVal <- unique(raw1)
            
            if (length(setdiff(rawVal, (expVal <- c("NA", "No", "Ys")))) == 0) {
                raw1 <- gsub("Yes", "Ys", raw1, fixed = TRUE)
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Me", "Circumstances")))) == 0) {
                raw1 <- gsub("Circumstances", "Cs", raw1, fixed = TRUE)
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Grrr people", "Yay people!")))) == 0) {
                raw1 <- gsub("Grrr people", "Gr", raw1, fixed = TRUE)
                raw1 <- gsub("Yay people!", "Yy", raw1, fixed = TRUE)                
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Idealist", "Pragmatist")))) == 0) {
                raw1 <- gsub("Idealist"  , "Id", raw1, fixed = TRUE)
                raw1 <- gsub("Pragmatist", "Pr", raw1, fixed = TRUE)                
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Private", "Public")))) == 0) {
                raw1 <- gsub("Private", "Pt", raw1, fixed = TRUE)
                raw1 <- gsub("Public" , "Pc", raw1, fixed = TRUE)                
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            }
            
            return(relevel(as.factor(raw1), ref = "NA"))
        }
        , args = c(qsn))

# If imputation of missing data is not working ...
# glbFeatsDerive[["FertilityRate.nonNA"]] <- list(
#     mapfn = function(FertilityRate, Region) {
#         RegionMdn <- tapply(FertilityRate, Region, FUN = median, na.rm = TRUE)
# 
#         retVal <- FertilityRate
#         retVal[is.na(FertilityRate)] <- RegionMdn[Region[is.na(FertilityRate)]]
#         return(retVal)
#     }
#     , args = c("FertilityRate", "Region"))
    
#     mapfn = function(HOSPI.COST) { return(cut(HOSPI.COST, 5, breaks = c(0, 100000, 200000, 300000, 900000), labels = NULL)) }     
#     mapfn = function(Rasmussen)  { return(ifelse(sign(Rasmussen) >= 0, 1, 0)) } 
#     mapfn = function(startprice) { return(startprice ^ (1/2)) }       
#     mapfn = function(startprice) { return(log(startprice)) }   
#     mapfn = function(startprice) { return(exp(-startprice / 20)) }
#     mapfn = function(startprice) { return(scale(log(startprice))) }     
#     mapfn = function(startprice) { return(sign(sprice.predict.diff) * (abs(sprice.predict.diff) ^ (1/10))) }        

    # factor      
#     mapfn = function(PropR) { return(as.factor(ifelse(PropR >= 0.5, "Y", "N"))) }
#     mapfn = function(productline, description) { as.factor(gsub(" ", "", productline)) }
#     mapfn = function(purpose) { return(relevel(as.factor(purpose), ref="all_other")) }
#     mapfn = function(raw) { tfr_raw <- as.character(cut(raw, 5)); 
#                             tfr_raw[is.na(tfr_raw)] <- "NA.my";
#                             return(as.factor(tfr_raw)) }
#     mapfn = function(startprice.log10) { return(cut(startprice.log10, 3)) }
#     mapfn = function(startprice.log10) { return(cut(sprice.predict.diff, c(-1000, -100, -10, -1, 0, 1, 10, 100, 1000))) }    

#     , args = c("<arg1>"))
    
    # multiple args
#     mapfn = function(id, date) { return(paste(as.character(id), as.character(date), sep = "#")) }        
#     mapfn = function(PTS, oppPTS) { return(PTS - oppPTS) }
#     mapfn = function(startprice.log10.predict, startprice) {
#                  return(spdiff <- (10 ^ startprice.log10.predict) - startprice) } 
#     mapfn = function(productline, description) { as.factor(
#         paste(gsub(" ", "", productline), as.numeric(nchar(description) > 0), sep = "*")) }
#     mapfn = function(.src, .pos) { 
#         return(paste(.src, sprintf("%04d", 
#                                    ifelse(.src == "Train", .pos, .pos - 7049)
#                                    ), sep = "#")) }       

# # If glbObsAll is not sorted in the desired manner
#     mapfn=function(Week) { return(coredata(lag(zoo(orderBy(~Week, glbObsAll)$ILI), -2, na.pad=TRUE))) }
#     mapfn=function(ILI) { return(coredata(lag(zoo(ILI), -2, na.pad=TRUE))) }
#     mapfn=function(ILI.2.lag) { return(log(ILI.2.lag)) }

# glbFeatsDerive[["<var1>"]] <- glbFeatsDerive[["<var2>"]]

# tst <- "descr.my"; args_lst <- NULL; for (arg in glbFeatsDerive[[tst]]$args) args_lst[[arg]] <- glbObsAll[, arg]; print(head(args_lst[[arg]])); print(head(drv_vals <- do.call(glbFeatsDerive[[tst]]$mapfn, args_lst))); 
# print(which_ix <- which(args_lst[[arg]] == 0.75)); print(drv_vals[which_ix]); 

glbFeatsDateTime <- list()
# Use OlsonNames() to enumerate supported time zones
# glbFeatsDateTime[["<DateTimeFeat>"]] <- 
#     c(format = "%Y-%m-%d %H:%M:%S" or "%m/%e/%y", timezone = "US/Eastern", impute.na = TRUE, 
#       last.ctg = FALSE, poly.ctg = FALSE)

glbFeatsPrice <- NULL # or c("<price_var>")

glbFeatsImage <- list() #list(<imageFeat> = list(patchSize = 10)) # if patchSize not specified, no patch computation

glbFeatsText <- list()
Sys.setlocale("LC_ALL", "C") # For english

## [1] "C/C/C/C/C/en_US.UTF-8"

#glbFeatsText[["<TextFeature>"]] <- list(NULL,
#   ,names = myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL, 
#       <comma-separated-screened-names>
#   ))))
#   ,rareWords = myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL, 
#       <comma-separated-nonSCOWL-words>
#   ))))
#)

# Text Processing Step: custom modifications not present in txt_munge -> use glbFeatsDerive
# Text Processing Step: universal modifications
glb_txt_munge_filenames_pfx <- "<projectId>_mytxt_"

# Text Processing Step: tolower
# Text Processing Step: myreplacePunctuation
# Text Processing Step: removeWords
glb_txt_stop_words <- list()
# Remember to use unstemmed words
if (length(glbFeatsText) > 0) {
    require(tm)
    require(stringr)

    glb_txt_stop_words[["<txt_var>"]] <- sort(myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
        # Remove any words from stopwords            
#         , setdiff(myreplacePunctuation(stopwords("english")), c("<keep_wrd1>", <keep_wrd2>"))
                                
        # Remove salutations
        ,"mr","mrs","dr","Rev"                                

        # Remove misc
        #,"th" # Happy [[:digit::]]+th birthday 

        # Remove terms present in Trn only or New only; search for "Partition post-stem"
        #   ,<comma-separated-terms>        

        # cor.y.train == NA
#         ,unlist(strsplit(paste(c(NULL
#           ,"<comma-separated-terms>"
#         ), collapse=",")

        # freq == 1; keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>

        # chisq.pval high (e.g. == 1); keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>

        # nzv.freqRatio high (e.g. >= glbFeatsNzvFreqMax); keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>        
                                            )))))
}
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^man", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
#glbObsAll[glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, 4866] > 0, c(glb_rsp_var, txtFeat)]

# To identify terms with a specific freq
#paste0(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], freq == 1)$term), collapse = ",")
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], freq <= 2)$term), collapse = ",")
#subset(glb_post_stem_words_terms_df_lst[[txtFeat]], term %in% c("zinger"))

# To identify terms with a specific freq & 
#   are not stemmed together later OR is value of color.fctr (e.g. gold)
#paste0(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], (freq == 1) & !(term %in% c("blacked","blemish","blocked","blocks","buying","cables","careful","carefully","changed","changing","chargers","cleanly","cleared","connect","connects","connected","contains","cosmetics","default","defaulting","defective","definitely","describe","described","devices","displays","drop","drops","engravement","excellant","excellently","feels","fix","flawlessly","frame","framing","gentle","gold","guarantee","guarantees","handled","handling","having","install","iphone","iphones","keeped","keeps","known","lights","line","lining","liquid","liquidation","looking","lots","manuals","manufacture","minis","most","mostly","network","networks","noted","opening","operated","performance","performs","person","personalized","photograph","physically","placed","places","powering","pre","previously","products","protection","purchasing","returned","rotate","rotation","running","sales","second","seconds","shipped","shuts","sides","skin","skinned","sticker","storing","thats","theres","touching","unusable","update","updates","upgrade","weeks","wrapped","verified","verify") ))$term), collapse = ",")

#print(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (freq <= 2)))
#glbObsAll[which(terms_mtrx[, 229] > 0), glbFeatsText]

# To identify terms with cor.y == NA
#orderBy(~-freq+term, subset(glb_post_stop_words_terms_df_lst[[txtFeat]], is.na(cor.y)))
#paste(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], is.na(cor.y))[, "term"]), collapse=",")
#orderBy(~-freq+term, subset(glb_post_stem_words_terms_df_lst[[txtFeat]], is.na(cor.y)))

# To identify terms with low cor.y.abs
#head(orderBy(~cor.y.abs+freq+term, subset(glb_post_stem_words_terms_df_lst[[txtFeat]], !is.na(cor.y))), 5)

# To identify terms with high chisq.pval
#subset(glb_post_stem_words_terms_df_lst[[txtFeat]], chisq.pval > 0.99)
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (chisq.pval > 0.99) & (freq <= 10))$term), collapse=",")
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (chisq.pval > 0.9))$term), collapse=",")
#head(orderBy(~-chisq.pval+freq+term, glb_post_stem_words_terms_df_lst[[txtFeat]]), 5)
#glbObsAll[glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, 68] > 0, glbFeatsText]
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^m", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])

# To identify terms with high nzv.freqRatio
#summary(glb_post_stem_words_terms_df_lst[[txtFeat]]$nzv.freqRatio)
#paste0(sort(setdiff(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (nzv.freqRatio >= glbFeatsNzvFreqMax) & (freq < 10) & (chisq.pval >= 0.05))$term, c( "128gb","3g","4g","gold","ipad1","ipad3","ipad4","ipadair2","ipadmini2","manufactur","spacegray","sprint","tmobil","verizon","wifion"))), collapse=",")

# To identify obs with a txt term
#tail(orderBy(~-freq+term, glb_post_stop_words_terms_df_lst[[txtFeat]]), 20)
#mydspObs(list(descr.my.contains="non"), cols=c("color", "carrier", "cellular", "storage"))
#grep("ever", dimnames(terms_stop_mtrx)$Terms)
#which(terms_stop_mtrx[, grep("ipad", dimnames(terms_stop_mtrx)$Terms)] > 0)
#glbObsAll[which(terms_stop_mtrx[, grep("16", dimnames(terms_stop_mtrx)$Terms)[1]] > 0), c(glbFeatsCategory, "storage", txtFeat)]

# Text Processing Step: screen for names # Move to glbFeatsText specs section in order of text processing steps
# glbFeatsText[["<txtFeat>"]]$names <- myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
#         # Person names for names screening
#         ,<comma-separated-list>
#         
#         # Company names
#         ,<comma-separated-list>
#                     
#         # Product names
#         ,<comma-separated-list>
#     ))))

# glbFeatsText[["<txtFeat>"]]$rareWords <- myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
#         # Words not in SCOWL db
#         ,<comma-separated-list>
#     ))))

# To identify char vectors post glbFeatsTextMap
#grep("six(.*)hour", glb_txt_chr_lst[[txtFeat]], ignore.case = TRUE, value = TRUE)
#grep("[S|s]ix(.*)[H|h]our", glb_txt_chr_lst[[txtFeat]], value = TRUE)

# To identify whether terms shd be synonyms
#orderBy(~term, glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^moder", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ])
# term_row_df <- glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^came$", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ]
# 
# cor(glb_post_stop_words_terms_mtrx_lst[[txtFeat]][glbObsAll$.lcn == "Fit", term_row_df$pos], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")

# To identify which stopped words are "close" to a txt term
#sort(glbFeatsCluster)

# Text Processing Step: stemDocument
# To identify stemmed txt terms
#glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^la$", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ]
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^con", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
#glbObsAll[which(terms_stem_mtrx[, grep("use", dimnames(terms_stem_mtrx)$Terms)[[1]]] > 0), c(glbFeatsId, "productline", txtFeat)]
#glbObsAll[which(TfIdf_stem_mtrx[, 191] > 0), c(glbFeatsId, glbFeatsCategory, txtFeat)]
#glbObsAll[which(glb_post_stop_words_terms_mtrx_lst[[txtFeat]][, 6165] > 0), c(glbFeatsId, glbFeatsCategory, txtFeat)]
#which(glbObsAll$UniqueID %in% c(11915, 11926, 12198))

# Text Processing Step: mycombineSynonyms
#   To identify which terms are associated with not -> combine "could not" & "couldn't"
#findAssocs(glb_full_DTM_lst[[txtFeat]], "not", 0.05)
#   To identify which synonyms should be combined
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^c", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
chk_comb_cor <- function(syn_lst) {
#     cor(terms_stem_mtrx[glbObsAll$.src == "Train", grep("^(damag|dent|ding)$", dimnames(terms_stem_mtrx)[[2]])], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
    print(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], term %in% syn_lst$syns))
    print(subset(get_corpus_terms(tm_map(glbFeatsTextCorpus[[txtFeat]], mycombineSynonyms, list(syn_lst), lazy=FALSE)), term == syn_lst$word))
#     cor(terms_stop_mtrx[glbObsAll$.src == "Train", grep("^(damage|dent|ding)$", dimnames(terms_stop_mtrx)[[2]])], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
#     cor(rowSums(terms_stop_mtrx[glbObsAll$.src == "Train", grep("^(damage|dent|ding)$", dimnames(terms_stop_mtrx)[[2]])]), glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
}
#chk_comb_cor(syn_lst=list(word="cabl",  syns=c("cabl", "cord")))
#chk_comb_cor(syn_lst=list(word="damag",  syns=c("damag", "dent", "ding")))
#chk_comb_cor(syn_lst=list(word="dent",  syns=c("dent", "ding")))
#chk_comb_cor(syn_lst=list(word="use",  syns=c("use", "usag")))

glbFeatsTextSynonyms <- list()
# list parsed to collect glbFeatsText[[<txtFeat>]]$vldTerms
# glbFeatsTextSynonyms[["Hdln.my"]] <- list(NULL
#     # people in places
#     , list(word = "australia", syns = c("australia", "australian"))
#     , list(word = "italy", syns = c("italy", "Italian"))
#     , list(word = "newyork", syns = c("newyork", "newyorker"))    
#     , list(word = "Pakistan", syns = c("Pakistan", "Pakistani"))    
#     , list(word = "peru", syns = c("peru", "peruvian"))
#     , list(word = "qatar", syns = c("qatar", "qatari"))
#     , list(word = "scotland", syns = c("scotland", "scotish"))
#     , list(word = "Shanghai", syns = c("Shanghai", "Shanzhai"))    
#     , list(word = "venezuela", syns = c("venezuela", "venezuelan"))    
# 
#     # companies - needs to be data dependent 
#     #   - e.g. ensure BNP in this experiment/feat always refers to BNPParibas
#         
#     # general synonyms
#     , list(word = "Create", syns = c("Create","Creator")) 
#     , list(word = "cute", syns = c("cute","cutest"))     
#     , list(word = "Disappear", syns = c("Disappear","Fadeout"))     
#     , list(word = "teach", syns = c("teach", "taught"))     
#     , list(word = "theater",  syns = c("theater", "theatre", "theatres")) 
#     , list(word = "understand",  syns = c("understand", "understood"))    
#     , list(word = "weak",  syns = c("weak", "weaken", "weaker", "weakest"))
#     , list(word = "wealth",  syns = c("wealth", "wealthi"))    
#     
#     # custom synonyms (phrases)
#     
#     # custom synonyms (names)
#                                       )
#glbFeatsTextSynonyms[["<txtFeat>"]] <- list(NULL
#     , list(word="<stem1>",  syns=c("<stem1>", "<stem1_2>"))
#                                       )

for (txtFeat in names(glbFeatsTextSynonyms))
    for (entryIx in 1:length(glbFeatsTextSynonyms[[txtFeat]])) {
        glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$word <-
            str_to_lower(glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$word)
        glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$syns <-
            str_to_lower(glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$syns)        
    }        

glbFeatsTextSeed <- 181
# tm options include: check tm::weightSMART 
glb_txt_terms_control <- list( # Gather model performance & run-time stats
                    # weighting = function(x) weightSMART(x, spec = "nnn")
                    # weighting = function(x) weightSMART(x, spec = "lnn")
                    # weighting = function(x) weightSMART(x, spec = "ann")
                    # weighting = function(x) weightSMART(x, spec = "bnn")
                    # weighting = function(x) weightSMART(x, spec = "Lnn")
                    # 
                    weighting = function(x) weightSMART(x, spec = "ltn") # default
                    # weighting = function(x) weightSMART(x, spec = "lpn")                    
                    # 
                    # weighting = function(x) weightSMART(x, spec = "ltc")                    
                    # 
                    # weighting = weightBin 
                    # weighting = weightTf 
                    # weighting = weightTfIdf # : default
                # termFreq selection criteria across obs: tm default: list(global=c(1, Inf))
                    , bounds = list(global = c(1, Inf)) 
                # wordLengths selection criteria: tm default: c(3, Inf)
                    , wordLengths = c(1, Inf) 
                              ) 

glb_txt_cor_var <- glb_rsp_var # : default # or c(<feat>)

# select one from c("union.top.val.cor", "top.cor", "top.val", default: "top.chisq", "sparse")
glbFeatsTextFilter <- "top.chisq" 
glbFeatsTextTermsMax <- rep(10, length(glbFeatsText)) # :default
names(glbFeatsTextTermsMax) <- names(glbFeatsText)

# Text Processing Step: extractAssoc
glbFeatsTextAssocCor <- rep(1, length(glbFeatsText)) # :default 
names(glbFeatsTextAssocCor) <- names(glbFeatsText)

# Remember to use stemmed terms
glb_important_terms <- list()

# Text Processing Step: extractPatterns (ngrams)
glbFeatsTextPatterns <- list()
#glbFeatsTextPatterns[[<txtFeat>>]] <- list()
#glbFeatsTextPatterns[[<txtFeat>>]] <- c(metropolitan.diary.colon = "Metropolitan Diary:")

# Have to set it even if it is not used
# Properties:
#   numrows(glb_feats_df) << numrows(glbObsFit
#   Select terms that appear in at least 0.2 * O(FP/FN(glbObsOOB)) ???
#       numrows(glbObsOOB) = 1.1 * numrows(glbObsNew) ???
glb_sprs_thresholds <- NULL # or c(<txtFeat1> = 0.988, <txtFeat2> = 0.970, <txtFeat3> = 0.970)

glbFctrMaxUniqVals <- 20 # default: 20
glb_impute_na_data <- FALSE # or TRUE
glb_mice_complete.seed <- 144 # or any integer

glbFeatsCluster <- c("YOB.Age.fctr", "Gender.fctr", "Income.fctr", 
                     # "Hhold.fctr",
                     "Edn.fctr",
                     paste(grep("^Q.", glbFeatsExclude, value = TRUE), "fctr", sep = ".")) # NULL : default or c("<feat1>", "<feat2>")
# glbFeatsCluster <- grep(paste0("[", 
#                         toupper(paste0(substr(glbFeatsText, 1, 1), collapse = "")),
#                                       "]\\.[PT]\\."), 
#                                names(glbObsAll), value = TRUE)

glb_cluster.seed <- 189 # or any integer
glbClusterEntropyVar <- NULL # c(glb_rsp_var, as.factor(cut(glb_rsp_var, 3)), default: NULL)
glbFeatsClusterVarsExclude <- TRUE # default FALSE

glb_interaction_only_feats <- NULL # : default or c(<parent_feat> = "<child_feat>")

glbFeatsNzvFreqMax <- 19 # 19 : caret default
glbFeatsNzvUniqMin <- 10 # 10 : caret default

glbRFESizes <- list()
#glbRFESizes[["mdlFamily"]] <- c(4, 8, 16, 32, 64, 67, 68, 69) # Accuracy@69/70 = 0.8258

glbObsFitOutliers <- list()
# If outliers.n >= 10; consider concatenation of interaction vars
# glbObsFitOutliers[["<mdlFamily>"]] <- c(NULL
#     is.na(.rstudent)
#     max(.rstudent)
#     is.na(.dffits)
#     .hatvalues >= 0.99        
#     -38,167,642 < minmax(.rstudent) < 49,649,823    
#     , <comma-separated-<glbFeatsId>>
#                                     )
glbObsTrnOutliers <- list()
glbObsTrnOutliers[["Final"]] <- union(glbObsFitOutliers[["All.X"]],
                                c(NULL
                                ))

# influence.measures: car::outlier; rstudent; dffits; hatvalues; dfbeta; dfbetas
#mdlId <- "All.X##rcv#glm"; obs_df <- fitobs_df
#mdlId <- "RFE.X.glm"; obs_df <- fitobs_df
#mdlId <- "Final.glm"; obs_df <- trnobs_df
#mdlId <- "CSM2.X.glm"; obs_df <- fitobs_df
#print(outliers <- car::outlierTest(glb_models_lst[[mdlId]]$finalModel))
#mdlIdFamily <- paste0(head(unlist(str_split(mdlId, "\\.")), -1), collapse="."); obs_df <- dplyr::filter_(obs_df, interp(~(!(var %in% glbObsFitOutliers[[mdlIdFamily]])), var = as.name(glbFeatsId))); model_diags_df <- cbind(obs_df, data.frame(.rstudent=stats::rstudent(glb_models_lst[[mdlId]]$finalModel)), data.frame(.dffits=stats::dffits(glb_models_lst[[mdlId]]$finalModel)), data.frame(.hatvalues=stats::hatvalues(glb_models_lst[[mdlId]]$finalModel)));print(summary(model_diags_df[, c(".rstudent",".dffits",".hatvalues")])); table(cut(model_diags_df$.hatvalues, breaks=c(0.00, 0.98, 0.99, 1.00)))

#print(subset(model_diags_df, is.na(.rstudent))[, glbFeatsId])
#print(model_diags_df[which.max(model_diags_df$.rstudent), ])
#print(subset(model_diags_df, is.na(.dffits))[, glbFeatsId])
#print(model_diags_df[which.min(model_diags_df$.dffits), ])
#print(subset(model_diags_df, .hatvalues > 0.99)[, glbFeatsId])
#dffits_df <- merge(dffits_df, outliers_df, by="row.names", all.x=TRUE); row.names(dffits_df) <- dffits_df$Row.names; dffits_df <- subset(dffits_df, select=-Row.names)
#dffits_df <- merge(dffits_df, glbObsFit, by="row.names", all.x=TRUE); row.names(dffits_df) <- dffits_df$Row.names; dffits_df <- subset(dffits_df, select=-Row.names)
#subset(dffits_df, !is.na(.Bonf.p))

#mdlId <- "CSM.X.glm"; vars <- myextract_actual_feats(row.names(orderBy(reformulate(c("-", paste0(mdlId, ".imp"))), myget_feats_imp(glb_models_lst[[mdlId]])))); 
#model_diags_df <- glb_get_predictions(model_diags_df, mdlId, glb_rsp_var)
#obs_ix <- row.names(model_diags_df) %in% names(outliers$rstudent)[1]
#obs_ix <- which(is.na(model_diags_df$.rstudent))
#obs_ix <- which(is.na(model_diags_df$.dffits))
#myplot_parcoord(obs_df=model_diags_df[, c(glbFeatsId, glbFeatsCategory, ".rstudent", ".dffits", ".hatvalues", glb_rsp_var, paste0(glb_rsp_var, mdlId), vars[1:min(20, length(vars))])], obs_ix=obs_ix, id_var=glbFeatsId, category_var=glbFeatsCategory)

#model_diags_df[row.names(model_diags_df) %in% names(outliers$rstudent)[c(1:2)], ]
#ctgry_diags_df <- model_diags_df[model_diags_df[, glbFeatsCategory] %in% c("Unknown#0"), ]
#myplot_parcoord(obs_df=ctgry_diags_df[, c(glbFeatsId, glbFeatsCategory, ".rstudent", ".dffits", ".hatvalues", glb_rsp_var, "startprice.log10.predict.RFE.X.glmnet", indepVar[1:20])], obs_ix=row.names(ctgry_diags_df) %in% names(outliers$rstudent)[1], id_var=glbFeatsId, category_var=glbFeatsCategory)
#table(glbObsFit[model_diags_df[, glbFeatsCategory] %in% c("iPad1#1"), "startprice.log10.cut.fctr"])
#glbObsFit[model_diags_df[, glbFeatsCategory] %in% c("iPad1#1"), c(glbFeatsId, "startprice")]

# No outliers & .dffits == NaN
#myplot_parcoord(obs_df=model_diags_df[, c(glbFeatsId, glbFeatsCategory, glb_rsp_var, "startprice.log10.predict.RFE.X.glmnet", indepVar[1:10])], obs_ix=seq(1:nrow(model_diags_df))[is.na(model_diags_df$.dffits)], id_var=glbFeatsId, category_var=glbFeatsCategory)

# Modify mdlId to (build & extract) "<FamilyId>#<Fit|Trn>#<caretMethod>#<preProc1.preProc2>#<samplingMethod>"
glb_models_lst <- list(); glb_models_df <- data.frame()

# Add xgboost algorithm

# Regression
if (glb_is_regression) {
    glbMdlMethods <- c(NULL
        # deterministic
            #, "lm", # same as glm
            , "glm", "bayesglm", "glmnet"
            , "rpart"
        # non-deterministic
            , "gbm", "rf" 
        # Unknown
            , "nnet" , "avNNet" # runs 25 models per cv sample for tunelength=5
            , "svmLinear", "svmLinear2"
            , "svmPoly" # runs 75 models per cv sample for tunelength=5
            , "svmRadial" 
            , "earth"
            , "bagEarth" # Takes a long time
            ,"xgbLinear","xgbTree"
        )
} else
# Classification - Add ada (auto feature selection)
    if (glb_is_binomial)
        glbMdlMethods <- c(NULL
        # deterministic                     
            , "bagEarth" # Takes a long time        
            , "glm", "bayesglm", "glmnet"
            , "nnet"
            , "rpart"
        # non-deterministic        
            , "gbm"
            , "avNNet" # runs 25 models per cv sample for tunelength=5      
            , "rf"
        # Unknown
            , "lda", "lda2"
                # svm models crash when predict is called -> internal to kernlab it should call predict without .outcome
            , "svmLinear", "svmLinear2"
            , "svmPoly" # runs 75 models per cv sample for tunelength=5
            , "svmRadial" 
            , "earth"
            ,"xgbLinear","xgbTree"
        ) else
        glbMdlMethods <- c(NULL
        # deterministic
            ,"glmnet"
        # non-deterministic 
            ,"rf"       
        # Unknown
            ,"gbm","rpart","xgbLinear","xgbTree"
        )

glbMdlFamilies <- list(); glb_mdl_feats_lst <- list()
# family: Choose from c("RFE.X", "CSM.X", "All.X", "Best.Interact")
#   methods: Choose from c(NULL, <method>, glbMdlMethods) 
#glbMdlFamilies[["RFE.X"]] <- c("glmnet", "glm") # non-NULL vector is mandatory
if (glb_is_classification && !glb_is_binomial)
    # glm does not work for multinomial
    glbMdlFamilies[["All.X"]] <- c("glmnet") else    
    glbMdlFamilies[["All.X"]] <- c("glmnet", "glm")

#glbMdlFamilies[["Best.Interact"]] <- "glmnet" # non-NULL vector is mandatory

# Check if interaction features make RFE better
# glbMdlFamilies[["CSM.X"]] <- setdiff(glbMdlMethods, c("lda", "lda2")) # crashing due to category:.clusterid ??? #c("glmnet", "glm") # non-NULL list is mandatory
# glb_mdl_feats_lst[["CSM.X"]] <- c(NULL
#     , <comma-separated-features-vector>
#                                   )
# dAFeats.CSM.X %<d-% c(NULL
#     # Interaction feats up to varImp(RFE.X.glmnet) >= 50
#     , <comma-separated-features-vector>
#     , setdiff(myextract_actual_feats(predictors(rfe_fit_results)), c(NULL
#                , <comma-separated-features-vector>
#                                                                       ))    
#                                   )
# glb_mdl_feats_lst[["CSM.X"]] <- "%<d-% dAFeats.CSM.X"

glbMdlFamilies[["Final"]] <- c(NULL) # NULL vector acceptable # c("glmnet", "glm")

glbMdlAllowParallel <- list()
#glbMdlAllowParallel[["Final##rcv#glmnet"]] <- FALSE
glbMdlAllowParallel[["All.X##rcv#glm"]] <- FALSE

# Check if tuning parameters make fit better; make it mdlFamily customizable ?
glbMdlTuneParams <- data.frame()
# When glmnet crashes at model$grid with error: ???
glmnetTuneParams <- rbind(data.frame()
                        ,data.frame(parameter = "alpha",  vals = "0.100 0.325 0.550 0.775 1.000")
                        ,data.frame(parameter = "lambda", vals = "9.342e-02")    
                        )
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams,
#                                cbind(data.frame(mdlId = "<mdlId>"),
#                                      glmnetTuneParams))

    #avNNet    
    #   size=[1] 3 5 7 9; decay=[0] 1e-04 0.001  0.01   0.1; bag=[FALSE]; RMSE=1.3300906 

    #bagEarth
    #   degree=1 [2] 3; nprune=64 128 256 512 [1024]; RMSE=0.6486663 (up)
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "bagEarth", parameter = "nprune", vals = "256")
#     ,data.frame(method = "bagEarth", parameter = "degree", vals = "2")    
# ))

    #earth 
    #   degree=[1]; nprune=2  [9] 17 25 33; RMSE=0.1334478
    
    #gbm 
    #   shrinkage=0.05 [0.10] 0.15 0.20 0.25; n.trees=100 150 200 [250] 300; interaction.depth=[1] 2 3 4 5; n.minobsinnode=[10]; RMSE=0.2008313     
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "gbm", parameter = "shrinkage", min = 0.05, max = 0.25, by = 0.05)
#     ,data.frame(method = "gbm", parameter = "n.trees", min = 100, max = 300, by = 50)
#     ,data.frame(method = "gbm", parameter = "interaction.depth", min = 1, max = 5, by = 1)
#     ,data.frame(method = "gbm", parameter = "n.minobsinnode", min = 10, max = 10, by = 10)
#     #seq(from=0.05,  to=0.25, by=0.05)
# ))

    #glmnet
    #   alpha=0.100 [0.325] 0.550 0.775 1.000; lambda=0.0005232693 0.0024288010 0.0112734954 [0.0523269304] 0.2428800957; RMSE=0.6164891
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "glmnet", parameter = "alpha", vals = "0.550 0.775 0.8875 0.94375 1.000")
#     ,data.frame(method = "glmnet", parameter = "lambda", vals = "9.858855e-05 0.0001971771 0.0009152152 0.0042480525 0.0197177130")    
# ))

    #nnet    
    #   size=3 5 [7] 9 11; decay=0.0001 0.001 0.01 [0.1] 0.2; RMSE=0.9287422
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "nnet", parameter = "size", vals = "3 5 7 9 11")
#     ,data.frame(method = "nnet", parameter = "decay", vals = "0.0001 0.0010 0.0100 0.1000 0.2000")    
# ))

    #rf # Don't bother; results are not deterministic
    #       mtry=2  35  68 [101] 134; RMSE=0.1339974
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "rf", parameter = "mtry", vals = "2 5 9 13 17")
# ))

    #rpart 
    #   cp=0.020 [0.025] 0.030 0.035 0.040; RMSE=0.1770237
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()    
#     ,data.frame(method = "rpart", parameter = "cp", vals = "0.004347826 0.008695652 0.017391304 0.021739130 0.034782609")
# ))
    
    #svmLinear
    #   C=0.01 0.05 [0.10] 0.50 1.00 2.00 3.00 4.00; RMSE=0.1271318; 0.1296718
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "svmLinear", parameter = "C", vals = "0.01 0.05 0.1 0.5 1")
# ))

    #svmLinear2    
    #   cost=0.0625 0.1250 [0.25] 0.50 1.00; RMSE=0.1276354 
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "svmLinear2", parameter = "cost", vals = "0.0625 0.125 0.25 0.5 1")
# ))

    #svmPoly    
    #   degree=[1] 2 3 4 5; scale=0.01 0.05 [0.1] 0.5 1; C=0.50 1.00 [2.00] 3.00 4.00; RMSE=0.1276130
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method="svmPoly", parameter="degree", min=1, max=5, by=1) #seq(1, 5, 1)
#     ,data.frame(method="svmPoly", parameter="scale", vals="0.01, 0.05, 0.1, 0.5, 1")
#     ,data.frame(method="svmPoly", parameter="C", vals="0.50, 1.00, 2.00, 3.00, 4.00")    
# ))

    #svmRadial
    #   sigma=[0.08674323]; C=0.25 0.50 1.00 [2.00] 4.00; RMSE=0.1614957
    
#glb2Sav(); all.equal(sav_models_df, glb_models_df)
    
glb_preproc_methods <- NULL
#     c("YeoJohnson", "center.scale", "range", "pca", "ica", "spatialSign")

# Baseline prediction model feature(s)
glb_Baseline_mdl_var <- NULL # or c("<feat>")

glbMdlMetric_terms <- NULL # or matrix(c(
#                               0,1,2,3,4,
#                               2,0,1,2,3,
#                               4,2,0,1,2,
#                               6,4,2,0,1,
#                               8,6,4,2,0
#                           ), byrow=TRUE, nrow=5)
glbMdlMetricSummary <- NULL # or "<metric_name>"
glbMdlMetricMaximize <- NULL # or FALSE (TRUE is not the default for both classification & regression) 
glbMdlMetricSummaryFn <- NULL # or function(data, lev=NULL, model=NULL) {
#     confusion_mtrx <- t(as.matrix(confusionMatrix(data$pred, data$obs)))
#     #print(confusion_mtrx)
#     #print(confusion_mtrx * glbMdlMetric_terms)
#     metric <- sum(confusion_mtrx * glbMdlMetric_terms) / nrow(data)
#     names(metric) <- glbMdlMetricSummary
#     return(metric)
# }

glbMdlCheckRcv <- FALSE # Turn it on when needed; otherwise takes long time
glb_rcv_n_folds <- 3 # or NULL
glb_rcv_n_repeats <- 3 # or NULL

glb_clf_proba_threshold <- NULL # 0.5

# Model selection criteria
if (glb_is_regression)
    glbMdlMetricsEval <- c("min.RMSE.OOB", "max.R.sq.OOB", "max.Adj.R.sq.fit", "min.RMSE.fit")
    #glbMdlMetricsEval <- c("min.RMSE.fit", "max.R.sq.fit", "max.Adj.R.sq.fit")    
if (glb_is_classification) {
    if (glb_is_binomial)
        glbMdlMetricsEval <- 
            c("max.Accuracy.OOB", "max.AUCROCR.OOB", "max.AUCpROC.OOB", "min.aic.fit", "max.Accuracy.fit") else        
        glbMdlMetricsEval <- c("max.Accuracy.OOB", "max.Kappa.OOB")
}

# select from NULL [no ensemble models], "auto" [all models better than MFO or Baseline], c(mdl_ids in glb_models_lst) [Typically top-rated models in auto]
glb_mdl_ensemble <- NULL
#     "%<d-% setdiff(mygetEnsembleAutoMdlIds(), 'CSM.X.rf')" 
#     c(<comma-separated-mdlIds>
#      )

# Only for classifications; for regressions remove "(.*)\\.prob" form the regex
# tmp_fitobs_df <- glbObsFit[, grep(paste0("^", gsub(".", "\\.", mygetPredictIds$value, fixed = TRUE), "CSM\\.X\\.(.*)\\.prob"), names(glbObsFit), value = TRUE)]; cor_mtrx <- cor(tmp_fitobs_df); cor_vctr <- sort(cor_mtrx[row.names(orderBy(~-Overall, varImp(glb_models_lst[["Ensemble.repeatedcv.glmnet"]])$imp))[1], ]); summary(cor_vctr); cor_vctr
#ntv.glm <- glm(reformulate(indepVar, glb_rsp_var), family = "binomial", data = glbObsFit)
#step.glm <- step(ntv.glm)

glbMdlSelId <- "All.X##rcv#glmnet" #select from c(NULL, "All.X##rcv#glmnet", "RFE.X##rcv#glmnet", <mdlId>)
glbMdlFinId <- NULL #select from c(NULL, glbMdlSelId)

glb_dsp_cols <- c(".pos", glbFeatsId, glbFeatsCategory, glb_rsp_var
#               List critical cols excl. above
                  )

# Output specs
# lclgetfltout_df <- function(obsOutFinDf) {
#     require(tidyr)
#     obsOutFinDf <- obsOutFinDf %>%
#         tidyr::separate("ImageId.x.y", c(".src", ".pos", "x", "y"), 
#                         sep = "#", remove = TRUE, extra = "merge")
#     # mnm prefix stands for max_n_mean
#     mnmout_df <- obsOutFinDf %>%
#         dplyr::group_by(.pos) %>%
#         #dplyr::top_n(1, Probability1) %>% # Score = 3.9426         
#         #dplyr::top_n(2, Probability1) %>% # Score = ???; weighted = 3.94254;         
#         #dplyr::top_n(3, Probability1) %>% # Score = 3.9418; weighted = 3.94169; 
#         dplyr::top_n(4, Probability1) %>% # Score = ???; weighted = 3.94149;        
#         #dplyr::top_n(5, Probability1) %>% # Score = 3.9421; weighted = 3.94178
#     
#         # dplyr::summarize(xMeanN = mean(as.numeric(x)), yMeanN = mean(as.numeric(y)))
#         # dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), Probability1), yMeanN = mean(as.numeric(y)))
#         # dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), c(Probability1, 0.2357323, 0.2336925)), yMeanN = mean(as.numeric(y)))    
#         # dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), c(Probability1)), yMeanN = mean(as.numeric(y)))
#         dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), c(Probability1)), 
#                          yMeanN = weighted.mean(as.numeric(y), c(Probability1)))  
#     
#     maxout_df <- obsOutFinDf %>%
#         dplyr::group_by(.pos) %>%
#         dplyr::summarize(maxProb1 = max(Probability1))
#     fltout_df <- merge(maxout_df, obsOutFinDf, 
#                        by.x = c(".pos", "maxProb1"), by.y = c(".pos", "Probability1"),
#                        all.x = TRUE)
#     fmnout_df <- merge(fltout_df, mnmout_df, 
#                        by.x = c(".pos"), by.y = c(".pos"),
#                        all.x = TRUE)
#     return(fmnout_df)
# }
glbObsOut <- list(NULL
        # glbFeatsId will be the first output column, by default
        ,vars = list()
#         ,mapFn = function(obsOutFinDf) {
#                   }
                  )
#obsOutFinDf <- savobsOutFinDf
# glbObsOut$mapFn <- function(obsOutFinDf) {
#     txfout_df <- dplyr::select(obsOutFinDf, -.pos.y) %>%
#         dplyr::mutate(
#             lunch     = levels(glbObsTrn[, "lunch"    ])[
#                        round(mean(as.numeric(glbObsTrn[, "lunch"    ])), 0)],
#             dinner    = levels(glbObsTrn[, "dinner"   ])[
#                        round(mean(as.numeric(glbObsTrn[, "dinner"   ])), 0)],
#             reserve   = levels(glbObsTrn[, "reserve"  ])[
#                        round(mean(as.numeric(glbObsTrn[, "reserve"  ])), 0)],
#             outdoor   = levels(glbObsTrn[, "outdoor"  ])[
#                        round(mean(as.numeric(glbObsTrn[, "outdoor"  ])), 0)],
#             expensive = levels(glbObsTrn[, "expensive"])[
#                        round(mean(as.numeric(glbObsTrn[, "expensive"])), 0)],
#             liquor    = levels(glbObsTrn[, "liquor"   ])[
#                        round(mean(as.numeric(glbObsTrn[, "liquor"   ])), 0)],
#             table     = levels(glbObsTrn[, "table"    ])[
#                        round(mean(as.numeric(glbObsTrn[, "table"    ])), 0)],
#             classy    = levels(glbObsTrn[, "classy"   ])[
#                        round(mean(as.numeric(glbObsTrn[, "classy"   ])), 0)],
#             kids      = levels(glbObsTrn[, "kids"     ])[
#                        round(mean(as.numeric(glbObsTrn[, "kids"     ])), 0)]
#                       )
#     
#     print("ObsNew output class tables:")
#     print(sapply(c("lunch","dinner","reserve","outdoor",
#                    "expensive","liquor","table",
#                    "classy","kids"), 
#                  function(feat) table(txfout_df[, feat], useNA = "ifany")))
#     
#     txfout_df <- txfout_df %>%
#         dplyr::mutate(labels = "") %>%
#         dplyr::mutate(labels = 
#     ifelse(lunch     != "-1", paste(labels, lunch    ), labels)) %>%
#         dplyr::mutate(labels = 
#     ifelse(dinner    != "-1", paste(labels, dinner   ), labels)) %>%
#         dplyr::mutate(labels = 
#     ifelse(reserve   != "-1", paste(labels, reserve  ), labels)) %>%
#         dplyr::mutate(labels = 
#     ifelse(outdoor   != "-1", paste(labels, outdoor  ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(expensive != "-1", paste(labels, expensive), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(liquor    != "-1", paste(labels, liquor   ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(table     != "-1", paste(labels, table    ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(classy    != "-1", paste(labels, classy   ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(kids      != "-1", paste(labels, kids     ), labels)) %>%
#         dplyr::select(business_id, labels)
#     return(txfout_df)
# }
#if (!is.null(glbObsOut$mapFn)) obsOutFinDf <- glbObsOut$mapFn(obsOutFinDf); print(head(obsOutFinDf))

glb_out_obs <- NULL # select from c(NULL : default to "new", "all", "new", "trn")

if (glb_is_classification && glb_is_binomial) {
    # glbObsOut$vars[["Probability1"]] <- 
    #     "%<d-% glbObsNew[, mygetPredictIds(glb_rsp_var, glbMdlFinId)$prob]" 
    # glbObsOut$vars[[glb_rsp_var_raw]] <-
    #     "%<d-% glb_map_rsp_var_to_raw(glbObsNew[,
    #                                         mygetPredictIds(glb_rsp_var, glbMdlFinId)$value])"
    glbObsOut$vars[["Predictions"]] <-
        "%<d-% glb_map_rsp_var_to_raw(glbObsNew[,
                                            mygetPredictIds(glb_rsp_var, glbMdlFinId)$value])"
} else {
#     glbObsOut$vars[[glbFeatsId]] <- 
#         "%<d-% as.integer(gsub('Test#', '', glbObsNew[, glbFeatsId]))"
    glbObsOut$vars[[glb_rsp_var]] <- 
        "%<d-% glbObsNew[, mygetPredictIds(glb_rsp_var, glbMdlFinId)$value]"
#     for (outVar in setdiff(glbFeatsExcludeLcl, glb_rsp_var_raw))
#         glbObsOut$vars[[outVar]] <- 
#             paste0("%<d-% mean(glbObsAll[, \"", outVar, "\"], na.rm = TRUE)")
}    
# glbObsOut$vars[[glb_rsp_var_raw]] <- glb_rsp_var_raw
# glbObsOut$vars[[paste0(head(unlist(strsplit(mygetPredictIds$value, "")), -1), collapse = "")]] <-

glbOutStackFnames <- NULL #: default
    # c("ebayipads_txt_assoc1_out_bid1_stack.csv") # manual stack
    # c("ebayipads_finmdl_bid1_out_nnet_1.csv") # universal stack

glbOut <- list(pfx = "Votes_Q_02_cluster_")
# lclImageSampleSeed <- 129
glbOutDataVizFname <- NULL # choose from c(NULL, "<projectId>_obsall.csv")


glbChunks <- list(labels = c("set_global_options_wd","set_global_options"
    ,"import.data","inspect.data","scrub.data","transform.data"
    ,"extract.features"
        ,"extract.features.datetime","extract.features.image","extract.features.price"
        ,"extract.features.text","extract.features.string"  
        ,"extract.features.end"
    ,"manage.missing.data","cluster.data","partition.data.training","select.features"
    ,"fit.models_0","fit.models_1","fit.models_2","fit.models_3"
    ,"fit.data.training_0","fit.data.training_1"
    ,"predict.data.new"         
    ,"display.session.info"))
# To ensure that all chunks in this script are in glbChunks
if (!is.null(chkChunksLabels <- knitr::all_labels()) && # knitr::all_labels() doesn't work in console runs
    !identical(chkChunksLabels, glbChunks$labels)) {
    print(sprintf("setdiff(chkChunksLabels, glbChunks$labels): %s", 
                  setdiff(chkChunksLabels, glbChunks$labels)))    
    print(sprintf("setdiff(glbChunks$labels, chkChunksLabels): %s", 
                  setdiff(glbChunks$labels, chkChunksLabels)))    
}

glbChunks[["first"]] <- "cluster.data" #default: script will load envir from previous chunk
glbChunks[["last"]] <- NULL #NULL #default: script will save envir at end of this chunk 
glbChunks[["inpFilePathName"]] <- "data/Votes_Q_01_cnk_manage.missing.data.RData"
#mysavChunk(glbOut$pfx, glbChunks[["last"]]) # called from myevlChunk

# Inspect max OOB FP
#chkObsOOB <- subset(glbObsOOB, !label.fctr.All.X..rcv.glmnet.is.acc)
#chkObsOOBFP <- subset(chkObsOOB, label.fctr.All.X..rcv.glmnet == "left_eye_center") %>% dplyr::mutate(Probability1 = label.fctr.All.X..rcv.glmnet.prob) %>% select(-.src, -.pos, -x, -y) %>% lclgetfltout_df() %>% mutate(obj.distance = (((as.numeric(x) - left_eye_center_x.int) ^ 2) + ((as.numeric(y) - left_eye_center_y.int) ^ 2)) ^ 0.5) %>% dplyr::top_n(5, obj.distance) %>% dplyr::top_n(5, -patch.cor)
#
#newImgObs <- glbObsNew[(glbObsNew$ImageId == "Test#0001"), ]; print(newImgObs[which.max(newImgObs$label.fctr.Final..rcv.glmnet.prob), ])
#OOBImgObs <- glbObsOOB[(glbObsOOB$ImageId == "Train#0003"), ]; print(OOBImgObs[which.max(OOBImgObs$label.fctr.All.X..rcv.glmnet.prob), ])

#load("Votes_Q_02_cnk_extract.features.end.RData", verbose = TRUE)
#mygetImage(which(glbObsAll[, glbFeatsId] == "Train#0003"), names(glbFeatsImage)[1], plot = TRUE, featHighlight = c("left_eye_center_x", "left_eye_center_y"), ovrlHighlight = c(66, 35))

# Depict process
glb_analytics_pn <- petrinet(name = "glb_analytics_pn",
                        trans_df = data.frame(id = 1:6,
    name = c("data.training.all","data.new",
           "model.selected","model.final",
           "data.training.all.prediction","data.new.prediction"),
    x=c(   -5,-5,-15,-25,-25,-35),
    y=c(   -5, 5,  0,  0, -5,  5)
                        ),
                        places_df=data.frame(id=1:4,
    name=c("bgn","fit.data.training.all","predict.data.new","end"),
    x=c(   -0,   -20,                    -30,               -40),
    y=c(    0,     0,                      0,                 0),
    M0=c(   3,     0,                      0,                 0)
                        ),
                        arcs_df = data.frame(
    begin = c("bgn","bgn","bgn",        
            "data.training.all","model.selected","fit.data.training.all",
            "fit.data.training.all","model.final",    
            "data.new","predict.data.new",
            "data.training.all.prediction","data.new.prediction"),
    end   = c("data.training.all","data.new","model.selected",
            "fit.data.training.all","fit.data.training.all","model.final",
            "data.training.all.prediction","predict.data.new",
            "predict.data.new","data.new.prediction",
            "end","end")
                        ))
#print(ggplot.petrinet(glb_analytics_pn))
print(ggplot.petrinet(glb_analytics_pn) + coord_flip())

## Loading required package: grid

glb_analytics_avl_objs <- NULL

glb_chunks_df <- myadd_chunk(NULL, 
                             ifelse(is.null(glbChunks$first), "import.data", glbChunks$first))

##          label step_major step_minor label_minor   bgn end elapsed
## 1 cluster.data          1          0           0 6.109  NA      NA

Step `1.0: cluster data`

chunk option: eval=

Step `1.0: cluster data`

```{r extract.features.image, cache=FALSE, echo=FALSE, fig.height=5, fig.width=5, eval=myevlChunk(glbChunks, glbOut$pfx)}

Step `1.0: cluster data`

`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx)} #`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx, keep = c(“glbFeatsCategory”,“glb_dsp_cols”))}

## Loading required package: proxy

## 
## Attaching package: 'proxy'

## The following objects are masked from 'package:stats':
## 
##     as.dist, dist

## The following object is masked from 'package:base':
## 
##     as.matrix

## Loading required package: dynamicTreeCut

## Loading required package: entropy

## Loading required package: tidyr

## Loading required package: ggdendro

## [1] "Clustering features: "
##               abs.cor.y
## Q98197.fctr  0.05493425
## Q113181.fctr 0.08087531
## Q115611.fctr 0.09044682
## Gender.fctr  0.10274009
## Q109244.fctr 0.12038125
## [1] "    .rnorm cor: -0.0078"
## [1] "  Clustering entropy measure: Party.fctr"
## [1] "glbObsAll Entropy: 0.6913"

## Loading required package: lazyeval

##   Hhold.fctr .clusterid Hhold.fctr.clusterid    R    D  .entropy .knt
## 1          N          1                  N_1  220  230 0.6929002  450
## 2        MKn          1                MKn_1  308  344 0.6916221  652
## 3        MKy          1                MKy_1  842  752 0.6915524 1594
## 4        PKn          1                PKn_1   49  131 0.5854566  180
## 5        PKy          1                PKy_1   26   35 0.6822232   61
## 6        SKn          1                SKn_1 1091 1340 0.6878923 2431
## 7        SKy          1                SKy_1   81  119 0.6749870  200
## [1] "glbObsAll$Hhold.fctr Entropy: 0.6859 (99.2186 pct)"

## [1] "Category: N"
## [1] "max distance(0.9799) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 2175    2711          D          N      (50,65]           N       >150K
## 3742    4664          D          N           NA           N           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 2175      Bcr           NA           NA           NA           NA
## 3742        N           NA          Yes           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           Pc          Yes          Yes           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           No          Yes          Yes      Science
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 2175           NA           NA           NA           NA           NA
## 3742          Yes  Study first           No           NA           No
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 2175           NA           NA          Yes           NA           NA
## 3742       Giving          Yes           NA          Yes           No
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 2175           NA           NA             NA           NA           NA
## 3742           Pr           No Standard hours           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           NA          Yes           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           NA          End           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           Me          Yes           No           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 2175           NA           NA          Yes           NA           NA
## 3742           No           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 2175   Technology           NA           NA           NA           NA
## 3742           NA           NA           No          Yes           No
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 2175          Yes   Supportive           NA           NA           NA
## 3742          Yes    Demanding           No           NA          Yes
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA     Cautious           NA           No           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA    In-person           No          Yes           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 2175           NA           NA           NA           NA           NA
## 3742          Yes           Yy          Yes          Yes           No
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           No          Yes          Yes           No
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 2175          Yes           No           NA           NA           NA
## 3742           No          Yes           No          Yes           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           NA          Yes          Yes           No
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 2175          Yes           NA          NA          NA          NA
## 3742          Yes          Yes        Nope          NA          No
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 2175          NA          NA          NA          NA          NA
## 3742         Yes          NA          NA          NA         Yes
##      Q98078.fctr Q96024.fctr
## 2175          NA          NA
## 3742          No          NA
## [1] "min distance(0.9403) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 1286    1591          R          N      (15,20]           M           N
## 2641    3283          R          N      (15,20]           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 1286      K12           NA           NA           NA           NA
## 2641        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 1286           NA           NA           NA           NA           No
## 2641           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 1286           No   Mysterious           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 1286           NA          Yes           NA           NA           NA
## 2641           NA          Yes          Yes          Yes          Yes
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 1286           NA           NA          NA          NA          NA
## 2641           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 1286          NA          NA          NA          NA          NA
## 2641          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 1286          NA          NA
## 2641          NA          NA
## [1] "Category: MKn"
## [1] "max distance(0.9773) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 2328    2894          D        MKn      (65,90]           M     75-100K
## 4767    5946          R        MKn      (65,90]           N           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 2328        N           NA           No           No           No
## 4767      Msr           No           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 2328           Pc          Yes          Yes           No           No
## 4767           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 2328          Yes          Yes           No          Yes      Science
## 4767           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 2328           No  Study first          Yes          Yes          Yes
## 4767           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 2328    Receiving          Yes           No           NA           NA
## 4767           NA           NA           No           No           No
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           Pr          Yes           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA          Yes           NA           NA          Yes
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 2328           NA           NA           NA           NA           NA
## 4767          Yes   Supportive           No          Mac          Yes
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA         Yes!           No        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           No    In-person           No           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 2328           NA           NA           NA           NA           NA
## 4767          Yes           Yy           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 2328           NA           NA           NA           NA           NA
## 4767     Optimist          Dad           NA          Yes          Yes
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 2328           NA          Yes      Check!          No          No
## 4767          Yes           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 2328         Yes         Yes         Yes          NA          NA
## 4767          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 2328          NA          NA
## 4767          NA          NA
## [1] "min distance(0.9482) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3363    4185          D        MKn      (30,35]           F    100-150K
## 4551    5680          D        MKn      (25,30]           F      25-50K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3363      Bcr           NA           NA           NA           NA
## 4551      Bcr           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 3363           NA           NA             NA           NA           NA
## 4551           NA           NA Standard hours           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA          Yes           NA           NA           No
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3363          Yes           NA           NA           NA           NA
## 4551          Yes     Cautious       Umm...           NA        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA          Yes           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA          Yes          Yes           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3363           NA           NA          NA          NA          NA
## 4551           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3363          NA          NA          NA          NA          NA
## 4551          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 3363          NA          NA
## 4551          NA          NA
## [1] "Category: MKy"
## [1] "max distance(0.9786) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3492    4349          R        MKy      (65,90]           F    100-150K
## 4106    5121          R        MKy           NA           N           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3492        N           NA          Yes           NA           NA
## 4106      Msr           NA           NA          Yes           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3492           Pt          Yes           NA           NA           NA
## 4106           Pt           NA           NA           No           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3492           NA           NA           NA           NA           NA
## 4106           NA           NA          Yes          Yes      Science
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3492           NA           NA           NA          Yes           NA
## 4106          Yes  Study first           No          Yes          Yes
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3492           NA           NA           NA           NA           NA
## 4106       Giving           NA          Yes           NA           NA
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 3492           NA           NA             NA   Hot headed          Yes
## 4106           NA           No Standard hours           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3492           NA           NA          Yes           No           NA
## 4106           NA          Yes          Yes          Yes           No
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3492         A.M.           NA           NA          Yes           NA
## 4106         A.M.          Yes           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3492           Me           NA           NA          Yes           NA
## 4106           NA           NA           No           NA          Yes
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3492          Yes           NA           NA           NA         Talk
## 4106           NA           NA           NA           NA        Tunes
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3492           NA           NA          Yes          Yes           NA
## 4106       People           NA          Yes           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3492           NA           NA           NA          Mac           NA
## 4106          Yes           NA           NA           NA           No
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3492           NA           NA       Umm...           NA        Space
## 4106           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3492           NA       Online           NA           NA          Yes
## 4106           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3492           NA           NA           NA           NA           NA
## 4106           No           NA          Yes           No           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3492          Yes           NA           NA           NA           NA
## 4106           NA           No          Yes          Yes           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3492           NA           NA           NA           NA          Own
## 4106           NA           NA          Yes           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3492    Pessimist           NA           NA           NA           NA
## 4106     Optimist           NA           No          Yes          Yes
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3492           NA           NA          NA          NA          NA
## 4106           NA           No      Check!          No          No
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3492          NA          NA          NA          NA          NA
## 4106         Yes          NA         Yes          NA         Yes
##      Q98078.fctr Q96024.fctr
## 3492          NA          NA
## 4106          NA         Yes
## [1] "min distance(0.9489) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 4998    6245          R        MKy      (15,20]           M      50-75K
## 6032    2409       <NA>        MKy      (35,40]           M        <25K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 4998        N           NA           NA           NA           NA
## 6032        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 4998           NA           NA           NA           NA      Science
## 6032           NA           NA           NA           NA          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 4998          Yes    Try first           NA          Yes          Yes
## 6032           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 4998       Giving           NA           NA           NA           NA
## 6032           NA           NA           No           No           No
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 4998           Pr           NA           NA  Cool headed           NA
## 6032           Id          Yes           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 4998        Right           No           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 4998           NA   Supportive           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 4998          Yes           NA           NA          Yes           NA
## 6032           No           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 4998          Yes           NA           NA           NA           NA
## 6032           NA           NA           NA           No           No
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA          Yes
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 4998           NA           NA          NA          NA          NA
## 6032          Yes          Yes        Nope          No         Yes
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 4998          NA          NA          NA          NA          NA
## 6032          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 4998          NA          NA
## 6032          NA          NA
## [1] "Category: PKn"
## [1] "max distance(0.9742) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 404      502          D        PKn      (40,50]           F     75-100K
## 6145    2930       <NA>        PKn      (50,65]           F      25-50K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 404       PhD           NA          Yes           NA           NA
## 6145        N           NA           NA          Yes           No
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 404            NA           NA           NA           NA           No
## 6145           Pc           No          Yes           No           No
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 404           Yes          Yes          Yes           NA           NA
## 6145          Yes           No          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 404            NA           NA           NA           NA           NA
## 6145          Yes    Try first          Yes          Yes          Yes
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 404            NA           NA           NA           NA           NA
## 6145       Giving           No           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 404            NA          Yes          Yes          Yes           NA
## 6145           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 404          P.M.           NA           NA           NA           No
## 6145           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 404            NA           NA           NA           No          Yes
## 6145           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 404            NA          Yes          Yes          Yes          Yes
## 6145           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 404            No           No           No           No         Rent
## 6145           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 404      Optimist          Mom           No          Yes           NA
## 6145           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 404           Yes           NA      Check!          No         Yes
## 6145           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 404          Yes          NA          NA         Yes         Yes
## 6145          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 404          Yes          No
## 6145          NA          NA
## [1] "min distance(0.9462) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 848     1046          D        PKn      (50,65]           M    100-150K
## 3463    4312          D        PKn      (20,25]           F      25-50K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 848         N           NA           NA           NA           NA
## 3463      HSD           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 848            NA           NA           NA           NA          Yes
## 3463           NA           NA          Yes          Mac          Yes
##      Q109244.fctr  Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 848           Yes Risk-friendly         Yes!           No        Space
## 3463          Yes Risk-friendly         Yes!           No        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 848            No    In-person          Yes           NA           NA
## 3463          Yes           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 848            NA           NA          NA          NA          NA
## 3463           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 848           NA          NA          NA          NA          NA
## 3463          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 848           NA          No
## 3463          NA          NA
## [1] "Category: PKy"
## [1] "max distance(0.9739) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 1561    1933          R        PKy      (50,65]           F       >150K
## 4384    5471          R        PKy      (30,35]           F    100-150K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 1561      Ast           NA           NA           NA           NA
## 4384      HSD          Yes          Yes          Yes           No
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           Pc           No          Yes           No          Yes
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 1561           NA           NA           NA           NA           NA
## 4384          Yes           No          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           No  Study first          Yes          Yes          Yes
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 1561           NA           NA           NA          Yes          Yes
## 4384       Giving          Yes           No           No          Yes
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 1561           Pr           No             NA  Cool headed           No
## 4384           Id           No Standard hours   Hot headed           No
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 1561        Happy           No          Yes           No           No
## 4384        Right           No           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 1561         A.M.          Yes        Start          Yes           NA
## 4384           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 1561           Cs          Yes           No           No          Yes
## 4384           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 1561          Yes   Mysterious           No           No           NA
## 4384           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 1561           NA           NA          Yes           No           No
## 4384           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 1561           No          Yes           No           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA          Yes           No          Yes         Rent
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 1561           NA           NA           NA          Yes          Yes
## 4384     Optimist          Dad          Yes          Yes           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 1561           No          Yes      Check!          No          No
## 4384           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 1561         Yes         Yes          No          No         Yes
## 4384          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 1561         Yes         Yes
## 4384          NA          No
## [1] "min distance(0.9528) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3332    4150          D        PKy      (25,30]           M      50-75K
## 6815    6244       <NA>        PKy      (40,50]           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3332      HSD           NA           NA           NA           NA
## 6815      Ast           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3332           Pc           No          Yes           NA           No
## 6815           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3332           No           No           No           NA      Science
## 6815           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3332           No           NA           NA           No           No
## 6815           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3332           NA           NA           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 3332           NA           NA    Odd hours  Cool headed           NA
## 6815           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3332        Right           NA           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3332           NA           NA           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3332           NA           NA           NA           NA           No
## 6815           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3332           No           NA           NA           No           NA
## 6815           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3332           NA           No          Yes          Yes           NA
## 6815           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3332           NA           NA           NA           NA           No
## 6815           NA           NA           NA           PC          Yes
##      Q109244.fctr  Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3332           No      Cautious           NA           No           NA
## 6815          Yes Risk-friendly       Umm...           No        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3332           NA    In-person          Yes           NA           NA
## 6815           No           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3332          Yes           Gr           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3332           NA           NA           NA           No           NA
## 6815           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3332          Yes           NA           No           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3332    Pessimist           NA           NA          Yes           NA
## 6815           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3332           No           NA          NA          NA          NA
## 6815           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3332          NA          NA          NA          NA          NA
## 6815          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 3332          NA          NA
## 6815          NA          NA
## [1] "Category: SKn"
## [1] "max distance(0.9784) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 5083    6347          R        SKn           NA           N       >150K
## 6482    4629       <NA>        SKn      (30,35]           N      50-75K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 5083      PhD           NA           NA           NA           NA
## 6482        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 5083           Pt           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           No
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 5083           NA           NA           NA           NA           NA
## 6482          Yes          Yes          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 5083          Yes           NA           NA           NA           NA
## 6482           No  Study first           No           No           No
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 5083           NA           NA           NA           NA           NA
## 6482    Receiving          Yes           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 5083           NA           NA           NA           NA           No
## 6482           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 5083           NA           NA           NA           NA          Yes
## 6482           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA          Yes          Yes
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA          Yes           No           No
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 5083           NA           NA           NA           NA           NA
## 6482          Yes           No          Yes           No           No
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           No           No          Yes          Yes         Rent
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 5083           NA           NA           NA           NA           NA
## 6482     Optimist          Mom          Yes           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 5083           NA           NA          NA          NA          NA
## 6482           NA          Yes      Check!          No          No
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 5083          NA          NA          NA          NA          NA
## 6482         Yes         Yes          No          No  Only-child
##      Q98078.fctr Q96024.fctr
## 5083          NA          NA
## 6482         Yes          NA
## [1] "min distance(0.9355) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 2712    3375          D        SKn      (25,30]           M        <25K
## 4692    5856          R        SKn      (25,30]           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 2712      HSD           NA           NA           NA           NA
## 4692      K12           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 2712           NA           NA           NA           NA           No
## 4692           NA           NA           No           PC          Yes
##      Q109244.fctr  Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 2712          Yes      Cautious           NA           NA           NA
## 4692          Yes Risk-friendly       Umm...           NA        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 2712           No           NA           NA           NA           NA
## 4692          Yes           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 2712           NA           NA          NA          NA          NA
## 4692           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 2712          NA          NA          NA          NA          NA
## 4692          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 2712          NA          NA
## 4692          NA          No
## [1] "Category: SKy"
## [1] "max distance(0.9771) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3012    3748          R        SKy      (50,65]           F     75-100K
## 5348    6679          D        SKy      (40,50]           N     75-100K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3012        N           NA           NA           NA           NA
## 5348      PhD          Yes           No          Yes           No
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           Pc           No          Yes           No           No
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3012           NA           NA           NA          Yes          Art
## 5348          Yes           No          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3012           No  Study first          Yes           No          Yes
## 5348           No           NA           NA          Yes           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3012       Giving           No          Yes           No          Yes
## 5348           NA           NA          Yes           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 3012           Pr           No           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3012           NA          Yes        Start          Yes          Yes
## 5348           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3012           Me           No           No           No           No
## 5348           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3012           No          TMI          Yes           No        Tunes
## 5348           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3012       People           NA           NA           NA           No
## 5348           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3012          Yes    Demanding           No           PC           NA
## 5348           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3012           NA          Yes          Yes          Yes           No
## 5348           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3012           NA           NA           NA           NA          Own
## 5348           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3012     Optimist          Dad           No          Yes           No
## 5348           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3012           No          Yes      Check!         Yes          No
## 5348           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3012          NA          NA          NA          NA          NA
## 5348          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 3012          NA         Yes
## 5348          NA          NA
## [1] "min distance(0.9500) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 4126    5146          D        SKy           NA           M           N
## 6840    6393       <NA>        SKy           NA           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 4126      HSD           NA           NA           NA           NA
## 6840        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA          Yes           No          Yes
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 4126           NA          Yes           NA           NA           NA
## 6840         P.M.          Yes          End          Yes          Yes
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 4126           NA           No           No           No          Yes
## 6840           Cs           No          Yes          Yes          Yes
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 4126          Yes           NA           NA           NA           NA
## 6840           No   Mysterious           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 4126           NA           NA          NA          NA          NA
## 6840           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 4126          NA          NA          NA          NA          NA
## 6840          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 4126          NA          NA
## 6840          NA          NA
##    Hhold.fctr .clusterid Hhold.fctr.clusterid   R   D  .entropy .knt
## 1           N          1                  N_1  95 106 0.6916489  201
## 2           N          2                  N_2  77  76 0.6931258  153
## 3           N          3                  N_3  48  48 0.6931472   96
## 4         MKn          1                MKn_1 205 194 0.6927671  399
## 5         MKn          2                MKn_2  75  89 0.6894991  164
## 6         MKn          3                MKn_3  28  61 0.6227371   89
## 7         MKy          1                MKy_1 620 438 0.6782774 1058
## 8         MKy          2                MKy_2 134 136 0.6931197  270
## 9         MKy          3                MKy_3  88 178 0.6347628  266
## 10        PKn          1                PKn_1  27  69 0.5941300   96
## 11        PKn          2                PKn_2  10  46 0.4692203   56
## 12        PKn          3                PKn_3  12  16 0.6829081   28
## 13        PKy          1                PKy_1   8   9 0.6914161   17
## 14        PKy          2                PKy_2  10   8 0.6869616   18
## 15        PKy          3                PKy_3   7   5 0.6791933   12
## 16        PKy          4                PKy_4   1  13 0.2573186   14
## 17        SKn          1                SKn_1 512 456 0.6914729  968
## 18        SKn          2                SKn_2 257 475 0.6481206  732
## 19        SKn          3                SKn_3 167 200 0.6890991  367
## 20        SKn          4                SKn_4 155 209 0.6821024  364
## 21        SKy          1                SKy_1  45  86 0.6433372  131
## 22        SKy          2                SKy_2  21  17 0.6875967   38
## 23        SKy          3                SKy_3  15  16 0.6926268   31
## [1] "glbObsAll$Hhold.fctr$.clusterid Entropy: 0.6726 (98.0608 pct)"

##                     label step_major step_minor label_minor     bgn
## 1            cluster.data          1          0           0   6.109
## 2 partition.data.training          2          0           0 129.165
##       end elapsed
## 1 129.164 123.055
## 2      NA      NA

Step `2.0: partition data training`

## [1] "partition.data.training chunk: setup: elapsed: 0.00 secs"

## Loading required package: sqldf

## Loading required package: gsubfn

## Loading required package: proto

## Loading required package: RSQLite

## Loading required package: DBI

## Loading required package: tcltk

## Loading required package: reshape2

## [1] "partition.data.training chunk: strata_mtrx complete: elapsed: 3.67 secs"
## [1] "partition.data.training chunk: obs_freq_df complete: elapsed: 3.67 secs"

## Loading required package: sampling

## 
## Attaching package: 'sampling'

## The following object is masked from 'package:caret':
## 
##     cluster

## [1] "lclgetMatrixCorrelation: duration: 41.944000 secs"
## [1] "cor of Fit vs. OOB: 1.0000"
## [1] "lclgetMatrixCorrelation: duration: 14.834000 secs"
## [1] "cor of New vs. OOB: 1.0000"
## [1] "lclgetMatrixCorrelation: duration: 51.606000 secs"
## [1] "cor of Fit vs. New: 1.0000"

## [1] "partition.data.training chunk: Fit/OOB partition complete: elapsed: 112.71 secs"

##     Party.Democrat Party.Republican Party.NA
##                 NA               NA     1392
## Fit           2357             2091       NA
## OOB            594              526       NA
##     Party.Democrat Party.Republican Party.NA
##                 NA               NA        1
## Fit      0.5299011        0.4700989       NA
## OOB      0.5303571        0.4696429       NA
##   Hhold.fctr .n.Fit .n.OOB .n.Tst .freqRatio.Fit .freqRatio.OOB
## 6        SKn   1920    511    638     0.43165468    0.456250000
## 2        MKy   1296    298    371     0.29136691    0.266071429
## 1        MKn    516    136    169     0.11600719    0.121428571
## 3          N    367     83    102     0.08250899    0.074107143
## 7        SKy    147     53     65     0.03304856    0.047321429
## 4        PKn    150     30     37     0.03372302    0.026785714
## 5        PKy     52      9     10     0.01169065    0.008035714
##   .freqRatio.Tst
## 6    0.458333333
## 2    0.266522989
## 1    0.121408046
## 3    0.073275862
## 7    0.046695402
## 4    0.026580460
## 5    0.007183908

## [1] "glbObsAll: "

## [1] 6960  221

## [1] "glbObsTrn: "

## [1] 5568  221

## [1] "glbObsFit: "

## [1] 4448  220

## [1] "glbObsOOB: "

## [1] 1120  220

## [1] "glbObsNew: "

## [1] 1392  220

## [1] "partition.data.training chunk: teardown: elapsed: 113.63 secs"

##                     label step_major step_minor label_minor     bgn
## 2 partition.data.training          2          0           0 129.165
## 3         select.features          3          0           0 242.864
##       end elapsed
## 2 242.864 113.699
## 3      NA      NA

Step `3.0: select features`

##                         cor.y exclude.as.feat    cor.y.abs cor.high.X
## Q109244.fctr     0.1203812469               1 0.1203812469         NA
## .clusterid       0.0984277178               1 0.0984277178         NA
## .clusterid.fctr  0.0984277178               0 0.0984277178         NA
## Hhold.fctr       0.0511386673               0 0.0511386673         NA
## Edn.fctr         0.0359295351               1 0.0359295351         NA
## Q101163.fctr     0.0295046473               1 0.0295046473         NA
## Q100689.fctr     0.0256915080               1 0.0256915080         NA
## Q98078.fctr      0.0256516490               1 0.0256516490         NA
## Q99716.fctr      0.0209286674               1 0.0209286674         NA
## Q120379.fctr     0.0206291292               1 0.0206291292         NA
## Q121699.fctr     0.0196933075               1 0.0196933075         NA
## Q105840.fctr     0.0195569165               1 0.0195569165         NA
## Q113583.fctr     0.0191894717               1 0.0191894717         NA
## Q115195.fctr     0.0174522586               1 0.0174522586         NA
## Q102089.fctr     0.0174087944               1 0.0174087944         NA
## Q98059.fctr      0.0171637755               1 0.0171637755         NA
## Q114386.fctr     0.0168013326               1 0.0168013326         NA
## Q100680.fctr     0.0157762454               1 0.0157762454         NA
## Q108342.fctr     0.0151842510               1 0.0151842510         NA
## Q111848.fctr     0.0141099384               1 0.0141099384         NA
## YOB.Age.fctr     0.0129198495               1 0.0129198495         NA
## Q118892.fctr     0.0125250379               1 0.0125250379         NA
## Q102687.fctr     0.0120079165               1 0.0120079165         NA
## Q115390.fctr     0.0119300319               1 0.0119300319         NA
## Q119851.fctr     0.0093381833               1 0.0093381833         NA
## Q114517.fctr     0.0084741753               1 0.0084741753         NA
## Q120012.fctr     0.0084652930               1 0.0084652930         NA
## Q109367.fctr     0.0080456026               1 0.0080456026         NA
## Q114961.fctr     0.0079206587               1 0.0079206587         NA
## Q121700.fctr     0.0067756198               1 0.0067756198         NA
## Q124122.fctr     0.0061257448               1 0.0061257448         NA
## Q111220.fctr     0.0055758571               1 0.0055758571         NA
## Q113992.fctr     0.0041479796               1 0.0041479796         NA
## Q121011.fctr     0.0037329030               1 0.0037329030         NA
## Q106042.fctr     0.0032327194               1 0.0032327194         NA
## Q116448.fctr     0.0031731051               1 0.0031731051         NA
## Q116601.fctr     0.0022379241               1 0.0022379241         NA
## Q104996.fctr     0.0012202806               1 0.0012202806         NA
## Q102906.fctr     0.0011540297               1 0.0011540297         NA
## Q113584.fctr     0.0011387024               1 0.0011387024         NA
## Q108950.fctr     0.0010567028               1 0.0010567028         NA
## Q102674.fctr     0.0009759844               1 0.0009759844         NA
## Q103293.fctr     0.0005915534               1 0.0005915534         NA
## Q112478.fctr     0.0001517248               1 0.0001517248         NA
## Q114748.fctr    -0.0008477228               1 0.0008477228         NA
## Q107491.fctr    -0.0014031814               1 0.0014031814         NA
## Q100562.fctr    -0.0017132769               1 0.0017132769         NA
## Q108617.fctr    -0.0024119725               1 0.0024119725         NA
## Q100010.fctr    -0.0024291540               1 0.0024291540         NA
## Q115602.fctr    -0.0027844465               1 0.0027844465         NA
## Q116953.fctr    -0.0029786716               1 0.0029786716         NA
## Q115610.fctr    -0.0035255582               1 0.0035255582         NA
## Q106997.fctr    -0.0041749086               1 0.0041749086         NA
## Q120978.fctr    -0.0044187616               1 0.0044187616         NA
## Q112512.fctr    -0.0056768212               1 0.0056768212         NA
## Q108343.fctr    -0.0060665340               1 0.0060665340         NA
## Q96024.fctr     -0.0069116541               1 0.0069116541         NA
## Q106389.fctr    -0.0077498918               1 0.0077498918         NA
## .rnorm          -0.0078039520               0 0.0078039520         NA
## Q108754.fctr    -0.0080847764               1 0.0080847764         NA
## Q98578.fctr     -0.0081164509               1 0.0081164509         NA
## Q101162.fctr    -0.0099412952               1 0.0099412952         NA
## Q115777.fctr    -0.0101315203               1 0.0101315203         NA
## Q99581.fctr     -0.0103662478               1 0.0103662478         NA
## Q124742.fctr    -0.0111642906               1 0.0111642906         NA
## Q116797.fctr    -0.0112749656               1 0.0112749656         NA
## Q112270.fctr    -0.0116157798               1 0.0116157798         NA
## YOB             -0.0116828198               1 0.0116828198         NA
## Q118237.fctr    -0.0117079669               1 0.0117079669         NA
## Q119650.fctr    -0.0125645475               1 0.0125645475         NA
## Q111580.fctr    -0.0132382335               1 0.0132382335         NA
## Q123464.fctr    -0.0136140083               1 0.0136140083         NA
## Q117193.fctr    -0.0138241599               1 0.0138241599         NA
## Q99982.fctr     -0.0139727928               1 0.0139727928         NA
## Q108856.fctr    -0.0140363785               1 0.0140363785         NA
## Q118233.fctr    -0.0147269325               1 0.0147269325         NA
## Q102289.fctr    -0.0155850393               1 0.0155850393         NA
## Q116197.fctr    -0.0158561766               1 0.0158561766         NA
## Income.fctr     -0.0159635458               1 0.0159635458         NA
## Q118232.fctr    -0.0171321152               1 0.0171321152         NA
## Q120194.fctr    -0.0172986920               1 0.0172986920         NA
## Q114152.fctr    -0.0175013163               1 0.0175013163         NA
## Q122770.fctr    -0.0194639697               1 0.0194639697         NA
## Q117186.fctr    -0.0198853672               1 0.0198853672         NA
## Q105655.fctr    -0.0198994078               1 0.0198994078         NA
## Q106993.fctr    -0.0207428635               1 0.0207428635         NA
## Q119334.fctr    -0.0226894034               1 0.0226894034         NA
## Q122120.fctr    -0.0229287700               1 0.0229287700         NA
## Q116441.fctr    -0.0237358205               1 0.0237358205         NA
## Q118117.fctr    -0.0253544150               1 0.0253544150         NA
## Q123621.fctr    -0.0255329743               1 0.0255329743         NA
## Q122769.fctr    -0.0259739146               1 0.0259739146         NA
## Q120650.fctr    -0.0270889067               1 0.0270889067         NA
## Q98869.fctr     -0.0276734114               1 0.0276734114         NA
## .pos            -0.0302037138               1 0.0302037138         NA
## USER_ID         -0.0302304868               1 0.0302304868         NA
## Q107869.fctr    -0.0304661021               1 0.0304661021         NA
## Q120014.fctr    -0.0318620439               1 0.0318620439         NA
## Q115899.fctr    -0.0324177950               1 0.0324177950         NA
## Q106388.fctr    -0.0341579350               1 0.0341579350         NA
## Q99480.fctr     -0.0344412239               1 0.0344412239         NA
## Q122771.fctr    -0.0348421015               1 0.0348421015         NA
## Q108855.fctr    -0.0370970211               1 0.0370970211         NA
## Q110740.fctr    -0.0380691243               1 0.0380691243         NA
## Q106272.fctr    -0.0400926462               1 0.0400926462         NA
## Q101596.fctr    -0.0409784077               1 0.0409784077         NA
## Q116881.fctr    -0.0416860293               1 0.0416860293         NA
## Q120472.fctr    -0.0462030674               1 0.0462030674         NA
## Q98197.fctr     -0.0549342527               1 0.0549342527         NA
## Q113181.fctr    -0.0808753072               1 0.0808753072         NA
## Q115611.fctr    -0.0904468203               1 0.0904468203         NA
## Gender.fctr     -0.1027400851               1 0.1027400851         NA
##                 freqRatio percentUnique zeroVar   nzv is.cor.y.abs.low
## Q109244.fctr     1.125916    0.05387931   FALSE FALSE            FALSE
## .clusterid       2.005590    0.07183908   FALSE FALSE            FALSE
## .clusterid.fctr  2.005590    0.07183908   FALSE FALSE            FALSE
## Hhold.fctr       1.525094    0.12571839   FALSE FALSE            FALSE
## Edn.fctr         1.392610    0.14367816   FALSE FALSE            FALSE
## Q101163.fctr     1.327394    0.05387931   FALSE FALSE            FALSE
## Q100689.fctr     1.029800    0.05387931   FALSE FALSE            FALSE
## Q98078.fctr      1.266595    0.05387931   FALSE FALSE            FALSE
## Q99716.fctr      1.328693    0.05387931   FALSE FALSE            FALSE
## Q120379.fctr     1.046326    0.05387931   FALSE FALSE            FALSE
## Q121699.fctr     1.507127    0.05387931   FALSE FALSE            FALSE
## Q105840.fctr     1.275362    0.05387931   FALSE FALSE            FALSE
## Q113583.fctr     1.102515    0.05387931   FALSE FALSE            FALSE
## Q115195.fctr     1.065496    0.05387931   FALSE FALSE            FALSE
## Q102089.fctr     1.055963    0.05387931   FALSE FALSE            FALSE
## Q98059.fctr      1.493810    0.05387931   FALSE FALSE            FALSE
## Q114386.fctr     1.092072    0.05387931   FALSE FALSE            FALSE
## Q100680.fctr     1.102386    0.05387931   FALSE FALSE            FALSE
## Q108342.fctr     1.048292    0.05387931   FALSE FALSE            FALSE
## Q111848.fctr     1.113602    0.05387931   FALSE FALSE            FALSE
## YOB.Age.fctr     1.005794    0.16163793   FALSE FALSE            FALSE
## Q118892.fctr     1.347380    0.05387931   FALSE FALSE            FALSE
## Q102687.fctr     1.256545    0.05387931   FALSE FALSE            FALSE
## Q115390.fctr     1.150505    0.05387931   FALSE FALSE            FALSE
## Q119851.fctr     1.244519    0.05387931   FALSE FALSE            FALSE
## Q114517.fctr     1.183374    0.05387931   FALSE FALSE            FALSE
## Q120012.fctr     1.047185    0.05387931   FALSE FALSE            FALSE
## Q109367.fctr     1.008571    0.05387931   FALSE FALSE            FALSE
## Q114961.fctr     1.250436    0.05387931   FALSE FALSE            FALSE
## Q121700.fctr     1.708221    0.05387931   FALSE FALSE             TRUE
## Q124122.fctr     1.412807    0.05387931   FALSE FALSE             TRUE
## Q111220.fctr     1.262849    0.05387931   FALSE FALSE             TRUE
## Q113992.fctr     1.267442    0.05387931   FALSE FALSE             TRUE
## Q121011.fctr     1.153676    0.05387931   FALSE FALSE             TRUE
## Q106042.fctr     1.247738    0.05387931   FALSE FALSE             TRUE
## Q116448.fctr     1.161031    0.05387931   FALSE FALSE             TRUE
## Q116601.fctr     1.394914    0.05387931   FALSE FALSE             TRUE
## Q104996.fctr     1.173840    0.05387931   FALSE FALSE             TRUE
## Q102906.fctr     1.053396    0.05387931   FALSE FALSE             TRUE
## Q113584.fctr     1.212486    0.05387931   FALSE FALSE             TRUE
## Q108950.fctr     1.103872    0.05387931   FALSE FALSE             TRUE
## Q102674.fctr     1.073412    0.05387931   FALSE FALSE             TRUE
## Q103293.fctr     1.122287    0.05387931   FALSE FALSE             TRUE
## Q112478.fctr     1.113648    0.05387931   FALSE FALSE             TRUE
## Q114748.fctr     1.051125    0.05387931   FALSE FALSE             TRUE
## Q107491.fctr     1.419021    0.05387931   FALSE FALSE             TRUE
## Q100562.fctr     1.217215    0.05387931   FALSE FALSE             TRUE
## Q108617.fctr     1.390618    0.05387931   FALSE FALSE             TRUE
## Q100010.fctr     1.268156    0.05387931   FALSE FALSE             TRUE
## Q115602.fctr     1.322302    0.05387931   FALSE FALSE             TRUE
## Q116953.fctr     1.039180    0.05387931   FALSE FALSE             TRUE
## Q115610.fctr     1.359695    0.05387931   FALSE FALSE             TRUE
## Q106997.fctr     1.177632    0.05387931   FALSE FALSE             TRUE
## Q120978.fctr     1.131963    0.05387931   FALSE FALSE             TRUE
## Q112512.fctr     1.299253    0.05387931   FALSE FALSE             TRUE
## Q108343.fctr     1.064910    0.05387931   FALSE FALSE             TRUE
## Q96024.fctr      1.144428    0.05387931   FALSE FALSE             TRUE
## Q106389.fctr     1.341307    0.05387931   FALSE FALSE             TRUE
## .rnorm           1.000000  100.00000000   FALSE FALSE            FALSE
## Q108754.fctr     1.008090    0.05387931   FALSE FALSE            FALSE
## Q98578.fctr      1.093556    0.05387931   FALSE FALSE            FALSE
## Q101162.fctr     1.103229    0.05387931   FALSE FALSE            FALSE
## Q115777.fctr     1.140288    0.05387931   FALSE FALSE            FALSE
## Q99581.fctr      1.375000    0.05387931   FALSE FALSE            FALSE
## Q124742.fctr     2.565379    0.05387931   FALSE FALSE            FALSE
## Q116797.fctr     1.009589    0.05387931   FALSE FALSE            FALSE
## Q112270.fctr     1.254284    0.05387931   FALSE FALSE            FALSE
## YOB              1.027559    1.41882184   FALSE FALSE            FALSE
## Q118237.fctr     1.088017    0.05387931   FALSE FALSE            FALSE
## Q119650.fctr     1.456978    0.05387931   FALSE FALSE            FALSE
## Q111580.fctr     1.024977    0.05387931   FALSE FALSE            FALSE
## Q123464.fctr     1.326681    0.05387931   FALSE FALSE            FALSE
## Q117193.fctr     1.140665    0.05387931   FALSE FALSE            FALSE
## Q99982.fctr      1.339380    0.05387931   FALSE FALSE            FALSE
## Q108856.fctr     1.080645    0.05387931   FALSE FALSE            FALSE
## Q118233.fctr     1.199142    0.05387931   FALSE FALSE            FALSE
## Q102289.fctr     1.033482    0.05387931   FALSE FALSE            FALSE
## Q116197.fctr     1.073778    0.05387931   FALSE FALSE            FALSE
## Income.fctr      1.256724    0.12571839   FALSE FALSE            FALSE
## Q118232.fctr     1.365812    0.05387931   FALSE FALSE            FALSE
## Q120194.fctr     1.016716    0.05387931   FALSE FALSE            FALSE
## Q114152.fctr     1.027617    0.05387931   FALSE FALSE            FALSE
## Q122770.fctr     1.008802    0.05387931   FALSE FALSE            FALSE
## Q117186.fctr     1.053878    0.05387931   FALSE FALSE            FALSE
## Q105655.fctr     1.079316    0.05387931   FALSE FALSE            FALSE
## Q106993.fctr     1.327392    0.05387931   FALSE FALSE            FALSE
## Q119334.fctr     1.081498    0.05387931   FALSE FALSE            FALSE
## Q122120.fctr     1.297443    0.05387931   FALSE FALSE            FALSE
## Q116441.fctr     1.019645    0.05387931   FALSE FALSE            FALSE
## Q118117.fctr     1.174006    0.05387931   FALSE FALSE            FALSE
## Q123621.fctr     1.466381    0.05387931   FALSE FALSE            FALSE
## Q122769.fctr     1.060606    0.05387931   FALSE FALSE            FALSE
## Q120650.fctr     1.896247    0.05387931   FALSE FALSE            FALSE
## Q98869.fctr      1.080860    0.05387931   FALSE FALSE            FALSE
## .pos             1.000000  100.00000000   FALSE FALSE            FALSE
## USER_ID          1.000000  100.00000000   FALSE FALSE            FALSE
## Q107869.fctr     1.211050    0.05387931   FALSE FALSE            FALSE
## Q120014.fctr     1.044944    0.05387931   FALSE FALSE            FALSE
## Q115899.fctr     1.197849    0.05387931   FALSE FALSE            FALSE
## Q106388.fctr     1.065033    0.05387931   FALSE FALSE            FALSE
## Q99480.fctr      1.225404    0.05387931   FALSE FALSE            FALSE
## Q122771.fctr     1.414753    0.05387931   FALSE FALSE            FALSE
## Q108855.fctr     1.273980    0.05387931   FALSE FALSE            FALSE
## Q110740.fctr     1.050779    0.05387931   FALSE FALSE            FALSE
## Q106272.fctr     1.116536    0.05387931   FALSE FALSE            FALSE
## Q101596.fctr     1.041667    0.05387931   FALSE FALSE            FALSE
## Q116881.fctr     1.010066    0.05387931   FALSE FALSE            FALSE
## Q120472.fctr     1.292633    0.05387931   FALSE FALSE            FALSE
## Q98197.fctr      1.129371    0.05387931   FALSE FALSE            FALSE
## Q113181.fctr     1.006354    0.05387931   FALSE FALSE            FALSE
## Q115611.fctr     1.194859    0.05387931   FALSE FALSE            FALSE
## Gender.fctr      1.561033    0.05387931   FALSE FALSE            FALSE

## Warning in myplot_scatter(plt_feats_df, "percentUnique", "freqRatio",
## colorcol_name = "nzv", : converting nzv to class:factor

## Warning: Removed 3 rows containing missing values (geom_point).

## Warning: Removed 3 rows containing missing values (geom_point).

## Warning: Removed 3 rows containing missing values (geom_point).

## [1] cor.y            exclude.as.feat  cor.y.abs        cor.high.X      
## [5] freqRatio        percentUnique    zeroVar          nzv             
## [9] is.cor.y.abs.low
## <0 rows> (or 0-length row.names)

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

## [1] "numeric data missing in glbObsAll: "
##        YOB Party.fctr 
##        415       1392 
## [1] "numeric data w/ 0s in glbObsAll: "
## named integer(0)
## [1] "numeric data w/ Infs in glbObsAll: "
## named integer(0)
## [1] "numeric data w/ NaNs in glbObsAll: "
## named integer(0)
## [1] "string data missing in glbObsAll: "
##          Gender          Income HouseholdStatus  EducationLevel 
##             143            1273             552            1067 
##           Party         Q124742         Q124122         Q123464 
##              NA            4340            3114            2912 
##         Q123621         Q122769         Q122770         Q122771 
##            3018            2778            2597            2579 
##         Q122120         Q121699         Q121700         Q120978 
##            2552            2279            2328            2303 
##         Q121011         Q120379         Q120650         Q120472 
##            2256            2361            2283            2433 
##         Q120194         Q120012         Q120014         Q119334 
##            2603            2344            2571            2477 
##         Q119851         Q119650         Q118892         Q118117 
##            2243            2374            2206            2342 
##         Q118232         Q118233         Q118237         Q117186 
##            3018            2659            2592            2845 
##         Q117193         Q116797         Q116881         Q116953 
##            2799            2771            2889            2848 
##         Q116601         Q116441         Q116448         Q116197 
##            2606            2684            2730            2657 
##         Q115602         Q115777         Q115610         Q115611 
##            2619            2785            2637            2443 
##         Q115899         Q115390         Q114961         Q114748 
##            2789            2860            2687            2462 
##         Q115195         Q114517         Q114386         Q113992 
##            2647            2567            2686            2502 
##         Q114152         Q113583         Q113584         Q113181 
##            2829            2632            2654            2576 
##         Q112478         Q112512         Q112270         Q111848 
##            2790            2676            2820            2449 
##         Q111580         Q111220         Q110740         Q109367 
##            2686            2563            2479            2624 
##         Q108950         Q109244         Q108855         Q108617 
##            2641            2731            3008            2696 
##         Q108856         Q108754         Q108342         Q108343 
##            3007            2770            2760            2736 
##         Q107869         Q107491         Q106993         Q106997 
##            2762            2667            2676            2702 
##         Q106272         Q106388         Q106389         Q106042 
##            2722            2818            2871            2762 
##         Q105840         Q105655         Q104996         Q103293 
##            2876            2612            2620            2674 
##         Q102906         Q102674         Q102687         Q102289 
##            2840            2864            2712            2790 
##         Q102089         Q101162         Q101163         Q101596 
##            2736            2816            2995            2824 
##         Q100689         Q100680         Q100562          Q99982 
##            2568            2787            2793            2871 
##         Q100010          Q99716          Q99581          Q99480 
##            2688            2790            2690            2700 
##          Q98869          Q98578          Q98059          Q98078 
##            2906            2867            2629            2945 
##          Q98197          Q96024            .lcn 
##            2836            2858            1392

## [1] "glb_feats_df:"

## [1] 112  12

##                    id exclude.as.feat rsp_var
## Party.fctr Party.fctr            TRUE    TRUE

##                    id       cor.y exclude.as.feat  cor.y.abs cor.high.X
## USER_ID       USER_ID -0.03023049            TRUE 0.03023049         NA
## Party.fctr Party.fctr          NA            TRUE         NA         NA
##            freqRatio percentUnique zeroVar   nzv is.cor.y.abs.low
## USER_ID            1           100   FALSE FALSE            FALSE
## Party.fctr        NA            NA      NA    NA               NA
##            interaction.feat shapiro.test.p.value rsp_var_raw id_var
## USER_ID                <NA>                   NA       FALSE   TRUE
## Party.fctr             <NA>                   NA          NA     NA
##            rsp_var
## USER_ID         NA
## Party.fctr    TRUE

## [1] "glb_feats_df vs. glbObsAll: "

## character(0)

## [1] "glbObsAll vs. glb_feats_df: "

## character(0)

##             label step_major step_minor label_minor     bgn     end
## 3 select.features          3          0           0 242.864 245.735
## 4      fit.models          4          0           0 245.736      NA
##   elapsed
## 3   2.871
## 4      NA

Step `4.0: fit models`

fit.models_0_chunk_df <- myadd_chunk(NULL, "fit.models_0_bgn", label.minor = "setup")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_0_bgn          1          0       setup 246.296  NA      NA

# load(paste0(glbOut$pfx, "dsk.RData"))

get_model_sel_frmla <- function() {
    model_evl_terms <- c(NULL)
    # min.aic.fit might not be avl
    lclMdlEvlCriteria <- 
        glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)]
    for (metric in lclMdlEvlCriteria)
        model_evl_terms <- c(model_evl_terms, 
                             ifelse(length(grep("max", metric)) > 0, "-", "+"), metric)
    if (glb_is_classification && glb_is_binomial)
        model_evl_terms <- c(model_evl_terms, "-", "opt.prob.threshold.OOB")
    model_sel_frmla <- as.formula(paste(c("~ ", model_evl_terms), collapse = " "))
    return(model_sel_frmla)
}

get_dsp_models_df <- function() {
    dsp_models_cols <- c("id", 
                    glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                    grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
    dsp_models_df <- 
        #orderBy(get_model_sel_frmla(), glb_models_df)[, c("id", glbMdlMetricsEval)]
        orderBy(get_model_sel_frmla(), glb_models_df)[, dsp_models_cols]    
    nCvMdl <- sapply(glb_models_lst, function(mdl) nrow(mdl$results))
    nParams <- sapply(glb_models_lst, function(mdl) ifelse(mdl$method == "custom", 0, 
        nrow(subset(modelLookup(mdl$method), parameter != "parameter"))))
    
#     nCvMdl <- nCvMdl[names(nCvMdl) != "avNNet"]
#     nParams <- nParams[names(nParams) != "avNNet"]    
    
    if (length(cvMdlProblems <- nCvMdl[nCvMdl <= nParams]) > 0) {
        print("Cross Validation issues:")
        warning("Cross Validation issues:")        
        print(cvMdlProblems)
    }
    
    pltMdls <- setdiff(names(nCvMdl), names(cvMdlProblems))
    pltMdls <- setdiff(pltMdls, names(nParams[nParams == 0]))
    
    # length(pltMdls) == 21
    png(paste0(glbOut$pfx, "bestTune.png"), width = 480 * 2, height = 480 * 4)
    grid.newpage()
    pushViewport(viewport(layout = grid.layout(ceiling(length(pltMdls) / 2.0), 2)))
    pltIx <- 1
    for (mdlId in pltMdls) {
        print(ggplot(glb_models_lst[[mdlId]], highBestTune = TRUE) + labs(title = mdlId),   
              vp = viewport(layout.pos.row = ceiling(pltIx / 2.0), 
                            layout.pos.col = ((pltIx - 1) %% 2) + 1))  
        pltIx <- pltIx + 1
    }
    dev.off()

    if (all(row.names(dsp_models_df) != dsp_models_df$id))
        row.names(dsp_models_df) <- dsp_models_df$id
    return(dsp_models_df)
}
#get_dsp_models_df()

if (glb_is_classification && glb_is_binomial && 
        (length(unique(glbObsFit[, glb_rsp_var])) < 2))
    stop("glbObsFit$", glb_rsp_var, ": contains less than 2 unique values: ",
         paste0(unique(glbObsFit[, glb_rsp_var]), collapse=", "))

max_cor_y_x_vars <- orderBy(~ -cor.y.abs, 
        subset(glb_feats_df, (exclude.as.feat == 0) & !nzv & !is.cor.y.abs.low & 
                                is.na(cor.high.X)))[1:2, "id"]
max_cor_y_x_vars <- max_cor_y_x_vars[!is.na(max_cor_y_x_vars)]
if (length(max_cor_y_x_vars) < 2)
    max_cor_y_x_vars <- union(max_cor_y_x_vars, ".pos")

if (!is.null(glb_Baseline_mdl_var)) {
    if ((max_cor_y_x_vars[1] != glb_Baseline_mdl_var) & 
        (glb_feats_df[glb_feats_df$id == max_cor_y_x_vars[1], "cor.y.abs"] > 
         glb_feats_df[glb_feats_df$id == glb_Baseline_mdl_var, "cor.y.abs"]))
        stop(max_cor_y_x_vars[1], " has a higher correlation with ", glb_rsp_var, 
             " than the Baseline var: ", glb_Baseline_mdl_var)
}

glb_model_type <- ifelse(glb_is_regression, "regression", "classification")
    
# Model specs
# c("id.prefix", "method", "type",
#   # trainControl params
#   "preProc.method", "cv.n.folds", "cv.n.repeats", "summary.fn",
#   # train params
#   "metric", "metric.maximize", "tune.df")

# Baseline
if (!is.null(glb_Baseline_mdl_var)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                            paste0("fit.models_0_", "Baseline"), major.inc = FALSE,
                                    label.minor = "mybaseln_classfr")
    ret_lst <- myfit_mdl(mdl_id="Baseline", 
                         model_method="mybaseln_classfr",
                        indepVar=glb_Baseline_mdl_var,
                        rsp_var=glb_rsp_var,
                        fit_df=glbObsFit, OOB_df=glbObsOOB)
}    

# Most Frequent Outcome "MFO" model: mean(y) for regression
#   Not using caret's nullModel since model stats not avl
#   Cannot use rpart for multinomial classification since it predicts non-MFO
if (glb_is_classification) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                                paste0("fit.models_0_", "MFO"), major.inc = FALSE,
                                        label.minor = "myMFO_classfr")

    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "MFO", type = glb_model_type, trainControl.method = "none",
        train.method = ifelse(glb_is_regression, "lm", "myMFO_classfr"))),
                            indepVar = ".rnorm", rsp_var = glb_rsp_var,
                            fit_df = glbObsFit, OOB_df = glbObsOOB)

        # "random" model - only for classification; 
        #   none needed for regression since it is same as MFO
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                                paste0("fit.models_0_", "Random"), major.inc = FALSE,
                                        label.minor = "myrandom_classfr")

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)    
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Random", type = glb_model_type, trainControl.method = "none",
        train.method = "myrandom_classfr")),
                        indepVar = ".rnorm", rsp_var = glb_rsp_var,
                        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

##              label step_major step_minor   label_minor     bgn     end
## 1 fit.models_0_bgn          1          0         setup 246.296 246.328
## 2 fit.models_0_MFO          1          1 myMFO_classfr 246.329      NA
##   elapsed
## 1   0.032
## 2      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: MFO###myMFO_classfr"
## [1] "    indepVar: .rnorm"
## [1] "myfit_mdl: setup complete: 0.484000 secs"
## Fitting parameter = none on full training set
## [1] "in MFO.Classifier$fit"
## [1] "unique.vals:"
## [1] R D
## Levels: R D
## [1] "unique.prob:"
## y
##         D         R 
## 0.5299011 0.4700989 
## [1] "MFO.val:"
## [1] "D"
## [1] "myfit_mdl: train complete: 0.986000 secs"
##             Length Class      Mode     
## unique.vals 2      factor     numeric  
## unique.prob 2      -none-     numeric  
## MFO.val     1      -none-     character
## x.names     1      -none-     character
## xNames      1      -none-     character
## problemType 1      -none-     character
## tuneValue   1      data.frame list     
## obsLevels   2      -none-     character
## [1] "myfit_mdl: train diagnostics complete: 0.988000 secs"

## Loading required namespace: pROC

## [1] "entr MFO.Classifier$predict"
## [1] "exit MFO.Classifier$predict"

## Loading required package: ROCR

## Loading required package: gplots

## 
## Attaching package: 'gplots'

## The following object is masked from 'package:stats':
## 
##     lowess

## [1] "in MFO.Classifier$prob"
##           R         D
## 1 0.5299011 0.4700989
## 2 0.5299011 0.4700989
## 3 0.5299011 0.4700989
## 4 0.5299011 0.4700989
## 5 0.5299011 0.4700989
## 6 0.5299011 0.4700989

##          Prediction
## Reference    R    D
##         R 2091    0
##         D 2357    0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##      0.4700989      0.0000000      0.4553427      0.4848945      0.5299011 
## AccuracyPValue  McnemarPValue 
##      1.0000000      0.0000000 
## [1] "entr MFO.Classifier$predict"
## [1] "exit MFO.Classifier$predict"
## [1] "in MFO.Classifier$prob"
##           R         D
## 1 0.5299011 0.4700989
## 2 0.5299011 0.4700989
## 3 0.5299011 0.4700989
## 4 0.5299011 0.4700989
## 5 0.5299011 0.4700989
## 6 0.5299011 0.4700989

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 5.900000 secs"
##                    id  feats max.nTuningRuns min.elapsedtime.everything
## 1 MFO###myMFO_classfr .rnorm               0                      0.491
##   min.elapsedtime.final max.AUCpROC.fit max.Sens.fit max.Spec.fit
## 1                 0.004             0.5            0            1
##   max.AUCROCR.fit opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1             0.5                    0.5       0.6395473        0.4700989
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.4553427             0.4848945             0
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1             0.5            0            1             0.5
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                    0.5       0.6391252        0.4696429
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.4400805             0.4993651             0
## [1] "myfit_mdl: exit: 5.910000 secs"
##                 label step_major step_minor      label_minor     bgn
## 2    fit.models_0_MFO          1          1    myMFO_classfr 246.329
## 3 fit.models_0_Random          1          2 myrandom_classfr 252.245
##       end elapsed
## 2 252.244   5.915
## 3      NA      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Random###myrandom_classfr"
## [1] "    indepVar: .rnorm"
## [1] "myfit_mdl: setup complete: 0.436000 secs"
## Fitting parameter = none on full training set
## [1] "myfit_mdl: train complete: 0.735000 secs"
##             Length Class      Mode     
## unique.vals 2      factor     numeric  
## unique.prob 2      table      numeric  
## xNames      1      -none-     character
## problemType 1      -none-     character
## tuneValue   1      data.frame list     
## obsLevels   2      -none-     character
## [1] "myfit_mdl: train diagnostics complete: 0.736000 secs"
## [1] "in Random.Classifier$prob"

##          Prediction
## Reference    R    D
##         R 2091    0
##         D 2357    0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##      0.4700989      0.0000000      0.4553427      0.4848945      0.5299011 
## AccuracyPValue  McnemarPValue 
##      1.0000000      0.0000000 
## [1] "in Random.Classifier$prob"

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 6.783000 secs"
##                          id  feats max.nTuningRuns
## 1 Random###myrandom_classfr .rnorm               0
##   min.elapsedtime.everything min.elapsedtime.final max.AUCpROC.fit
## 1                      0.291                 0.003       0.4942483
##   max.Sens.fit max.Spec.fit max.AUCROCR.fit opt.prob.threshold.fit
## 1    0.4619799    0.5265168       0.5073101                   0.55
##   max.f.score.fit max.Accuracy.fit max.AccuracyLower.fit
## 1       0.6395473        0.4700989             0.4553427
##   max.AccuracyUpper.fit max.Kappa.fit max.AUCpROC.OOB max.Sens.OOB
## 1             0.4848945             0        0.523569          0.5
##   max.Spec.OOB max.AUCROCR.OOB opt.prob.threshold.OOB max.f.score.OOB
## 1     0.547138       0.5191202                   0.55       0.6391252
##   max.Accuracy.OOB max.AccuracyLower.OOB max.AccuracyUpper.OOB
## 1        0.4696429             0.4400805             0.4993651
##   max.Kappa.OOB
## 1             0
## [1] "myfit_mdl: exit: 6.796000 secs"

# Max.cor.Y
#   Check impact of cv
#       rpart is not a good candidate since caret does not optimize cp (only tuning parameter of rpart) well
fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                        paste0("fit.models_0_", "Max.cor.Y.rcv.*X*"), major.inc = FALSE,
                                    label.minor = "glmnet")

##                            label step_major step_minor      label_minor
## 3            fit.models_0_Random          1          2 myrandom_classfr
## 4 fit.models_0_Max.cor.Y.rcv.*X*          1          3           glmnet
##       bgn     end elapsed
## 3 252.245 259.052   6.807
## 4 259.053      NA      NA

ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
    id.prefix = "Max.cor.Y.rcv.1X1", type = glb_model_type, trainControl.method = "none",
    train.method = "glmnet")),
                    indepVar = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                    fit_df = glbObsFit, OOB_df = glbObsOOB)

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Max.cor.Y.rcv.1X1###glmnet"
## [1] "    indepVar: .clusterid.fctr,Hhold.fctr"
## [1] "myfit_mdl: setup complete: 0.695000 secs"

## Loading required package: glmnet

## Loading required package: Matrix

## 
## Attaching package: 'Matrix'

## The following object is masked from 'package:tidyr':
## 
##     expand

## Loaded glmnet 2.0-5

## Fitting alpha = 0.1, lambda = 0.000918 on full training set
## [1] "myfit_mdl: train complete: 1.528000 secs"

##             Length Class      Mode     
## a0           51    -none-     numeric  
## beta        459    dgCMatrix  S4       
## df           51    -none-     numeric  
## dim           2    -none-     numeric  
## lambda       51    -none-     numeric  
## dev.ratio    51    -none-     numeric  
## nulldev       1    -none-     numeric  
## npasses       1    -none-     numeric  
## jerr          1    -none-     numeric  
## offset        1    -none-     logical  
## classnames    2    -none-     character
## call          5    -none-     call     
## nobs          1    -none-     numeric  
## lambdaOpt     1    -none-     numeric  
## xNames        9    -none-     character
## problemType   1    -none-     character
## tuneValue     2    data.frame list     
## obsLevels     2    -none-     character
## [1] "min lambda > lambdaOpt:"
##      (Intercept) .clusterid.fctr2 .clusterid.fctr3 .clusterid.fctr4 
##       -0.1892506        0.4329138        0.4635477        0.4269151 
##    Hhold.fctrMKn    Hhold.fctrMKy    Hhold.fctrPKn    Hhold.fctrPKy 
##        0.1201887       -0.1244717        1.1421040        0.1608926 
##    Hhold.fctrSKn    Hhold.fctrSKy 
##        0.1515067        0.5006923 
## [1] "max lambda < lambdaOpt:"
## [1] "Feats mismatch between coefs_left & rght:"
##  [1] "(Intercept)"      ".clusterid.fctr2" ".clusterid.fctr3"
##  [4] ".clusterid.fctr4" "Hhold.fctrMKn"    "Hhold.fctrMKy"   
##  [7] "Hhold.fctrPKn"    "Hhold.fctrPKy"    "Hhold.fctrSKn"   
## [10] "Hhold.fctrSKy"   
## [1] "myfit_mdl: train diagnostics complete: 1.639000 secs"

##          Prediction
## Reference    R    D
##         R 2056   35
##         D 2242  115
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##     0.48808453     0.03025076     0.47329434     0.50289038     0.52990108 
## AccuracyPValue  McnemarPValue 
##     0.99999999     0.00000000

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 6.107000 secs"
##                           id                      feats max.nTuningRuns
## 1 Max.cor.Y.rcv.1X1###glmnet .clusterid.fctr,Hhold.fctr               0
##   min.elapsedtime.everything min.elapsedtime.final max.AUCpROC.fit
## 1                      0.823                 0.074       0.5754816
##   max.Sens.fit max.Spec.fit max.AUCROCR.fit opt.prob.threshold.fit
## 1     0.555715    0.5952482       0.4065966                    0.7
##   max.f.score.fit max.Accuracy.fit max.AccuracyLower.fit
## 1       0.6436062        0.4880845             0.4732943
##   max.AccuracyUpper.fit max.Kappa.fit max.AUCpROC.OOB max.Sens.OOB
## 1             0.5028904    0.03025076       0.5546626    0.5285171
##   max.Spec.OOB max.AUCROCR.OOB opt.prob.threshold.OOB max.f.score.OOB
## 1    0.5808081       0.4525163                   0.85       0.6391252
##   max.Accuracy.OOB max.AccuracyLower.OOB max.AccuracyUpper.OOB
## 1        0.4696429             0.4400805             0.4993651
##   max.Kappa.OOB
## 1             0
## [1] "myfit_mdl: exit: 6.120000 secs"

if (glbMdlCheckRcv) {
    # rcv_n_folds == 1 & rcv_n_repeats > 1 crashes
    for (rcv_n_folds in seq(3, glb_rcv_n_folds + 2, 2))
        for (rcv_n_repeats in seq(1, glb_rcv_n_repeats + 2, 2)) {
            
            # Experiment specific code to avoid caret crash
    #         lcl_tune_models_df <- rbind(data.frame()
    #                             ,data.frame(method = "glmnet", parameter = "alpha", 
    #                                         vals = "0.100 0.325 0.550 0.775 1.000")
    #                             ,data.frame(method = "glmnet", parameter = "lambda",
    #                                         vals = "9.342e-02")    
    #                                     )
            
            ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst =
                list(
                id.prefix = paste0("Max.cor.Y.rcv.", rcv_n_folds, "X", rcv_n_repeats), 
                type = glb_model_type, 
    # tune.df = lcl_tune_models_df,            
                trainControl.method = "repeatedcv",
                trainControl.number = rcv_n_folds, 
                trainControl.repeats = rcv_n_repeats,
                trainControl.classProbs = glb_is_classification,
                trainControl.summaryFunction = glbMdlMetricSummaryFn,
                train.method = "glmnet", train.metric = glbMdlMetricSummary, 
                train.maximize = glbMdlMetricMaximize)),
                                indepVar = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                                fit_df = glbObsFit, OOB_df = glbObsOOB)
        }
    # Add parallel coordinates graph of glb_models_df[, glbMdlMetricsEval] to evaluate cv parameters
    tmp_models_cols <- c("id", "max.nTuningRuns",
                        glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                        grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
    print(myplot_parcoord(obs_df = subset(glb_models_df, 
                                          grepl("Max.cor.Y.rcv.", id, fixed = TRUE), 
                                            select = -feats)[, tmp_models_cols],
                          id_var = "id"))
}
        
# Useful for stacking decisions
# fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
#                     paste0("fit.models_0_", "Max.cor.Y[rcv.1X1.cp.0|]"), major.inc = FALSE,
#                                     label.minor = "rpart")
# 
# ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
#     id.prefix = "Max.cor.Y.rcv.1X1.cp.0", type = glb_model_type, trainControl.method = "none",
#     train.method = "rpart",
#     tune.df=data.frame(method="rpart", parameter="cp", min=0.0, max=0.0, by=0.1))),
#                     indepVar=max_cor_y_x_vars, rsp_var=glb_rsp_var, 
#                     fit_df=glbObsFit, OOB_df=glbObsOOB)

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)
# if (glb_is_regression || glb_is_binomial) # For multinomials this model will be run next by default
ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                        id.prefix = "Max.cor.Y", 
                        type = glb_model_type, trainControl.method = "repeatedcv",
                        trainControl.number = glb_rcv_n_folds, 
                        trainControl.repeats = glb_rcv_n_repeats,
                        trainControl.classProbs = glb_is_classification,
                        trainControl.summaryFunction = glbMdlMetricSummaryFn,
                        trainControl.allowParallel = glbMdlAllowParallel,                        
                        train.metric = glbMdlMetricSummary, 
                        train.maximize = glbMdlMetricMaximize,    
                        train.method = "rpart")),
                    indepVar = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                    fit_df = glbObsFit, OOB_df = glbObsOOB)

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Max.cor.Y##rcv#rpart"
## [1] "    indepVar: .clusterid.fctr,Hhold.fctr"
## [1] "myfit_mdl: setup complete: 0.717000 secs"

## Loading required package: rpart

## Aggregating results
## Selecting tuning parameters
## Fitting cp = 0.000717 on full training set
## [1] "myfit_mdl: train complete: 2.496000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = "Max.cor.Y", : model's bestTune found at an extreme of
## tuneGrid for parameter: cp

## Loading required package: rpart.plot

## Call:
## rpart(formula = .outcome ~ ., control = list(minsplit = 20, minbucket = 7, 
##     cp = 0, maxcompete = 4, maxsurrogate = 5, usesurrogate = 2, 
##     surrogatestyle = 0, maxdepth = 30, xval = 0))
##   n= 4448 
## 
##             CP nsplit rel error
## 1 0.0516499283      0 1.0000000
## 2 0.0291726447      1 0.9483501
## 3 0.0042085127      2 0.9191774
## 4 0.0023912004      7 0.8981349
## 5 0.0007173601      9 0.8933525
## 
## Variable importance
## .clusterid.fctr3    Hhold.fctrMKy    Hhold.fctrPKn    Hhold.fctrSKn 
##               25               21               15               14 
## .clusterid.fctr2    Hhold.fctrSKy    Hhold.fctrMKn .clusterid.fctr4 
##                8                6                6                5 
##    Hhold.fctrPKy 
##                1 
## 
## Node number 1: 4448 observations,    complexity param=0.05164993
##   predicted class=D  expected loss=0.4700989  P(node) =1
##     class counts:  2091  2357
##    probabilities: 0.470 0.530 
##   left son=2 (1296 obs) right son=3 (3152 obs)
##   Primary splits:
##       Hhold.fctrMKy    < 0.5 to the right, improve=18.734760, (0 missing)
##       Hhold.fctrPKn    < 0.5 to the left,  improve=17.404310, (0 missing)
##       .clusterid.fctr2 < 0.5 to the left,  improve=11.862420, (0 missing)
##       .clusterid.fctr3 < 0.5 to the left,  improve= 6.520434, (0 missing)
##       Hhold.fctrSKn    < 0.5 to the left,  improve= 4.878051, (0 missing)
##   Surrogate splits:
##       Hhold.fctrSKn < 0.5 to the left,  agree=0.723, adj=0.049, (0 split)
## 
## Node number 2: 1296 observations,    complexity param=0.02917264
##   predicted class=R  expected loss=0.4583333  P(node) =0.2913669
##     class counts:   702   594
##    probabilities: 0.542 0.458 
##   left son=4 (1087 obs) right son=5 (209 obs)
##   Primary splits:
##       .clusterid.fctr3 < 0.5 to the left,  improve=17.5394500, (0 missing)
##       .clusterid.fctr2 < 0.5 to the left,  improve= 0.4996698, (0 missing)
## 
## Node number 3: 3152 observations,    complexity param=0.004208513
##   predicted class=D  expected loss=0.4406726  P(node) =0.7086331
##     class counts:  1389  1763
##    probabilities: 0.441 0.559 
##   left son=6 (3002 obs) right son=7 (150 obs)
##   Primary splits:
##       Hhold.fctrPKn    < 0.5 to the left,  improve=13.541280, (0 missing)
##       .clusterid.fctr2 < 0.5 to the left,  improve= 8.671724, (0 missing)
##       Hhold.fctrMKn    < 0.5 to the right, improve= 1.279129, (0 missing)
##       Hhold.fctrSKy    < 0.5 to the left,  improve= 1.099846, (0 missing)
##       .clusterid.fctr4 < 0.5 to the left,  improve= 1.089559, (0 missing)
## 
## Node number 4: 1087 observations
##   predicted class=R  expected loss=0.4222631  P(node) =0.2443795
##     class counts:   628   459
##    probabilities: 0.578 0.422 
## 
## Node number 5: 209 observations
##   predicted class=D  expected loss=0.354067  P(node) =0.04698741
##     class counts:    74   135
##    probabilities: 0.354 0.646 
## 
## Node number 6: 3002 observations,    complexity param=0.004208513
##   predicted class=D  expected loss=0.4510326  P(node) =0.6749101
##     class counts:  1354  1648
##    probabilities: 0.451 0.549 
##   left son=12 (2136 obs) right son=13 (866 obs)
##   Primary splits:
##       .clusterid.fctr2 < 0.5 to the left,  improve=7.0467130, (0 missing)
##       .clusterid.fctr4 < 0.5 to the left,  improve=1.7360290, (0 missing)
##       Hhold.fctrSKy    < 0.5 to the left,  improve=1.5182500, (0 missing)
##       Hhold.fctrSKn    < 0.5 to the left,  improve=0.6487705, (0 missing)
##       Hhold.fctrMKn    < 0.5 to the right, improve=0.5941805, (0 missing)
## 
## Node number 7: 150 observations
##   predicted class=D  expected loss=0.2333333  P(node) =0.03372302
##     class counts:    35   115
##    probabilities: 0.233 0.767 
## 
## Node number 12: 2136 observations,    complexity param=0.004208513
##   predicted class=D  expected loss=0.4728464  P(node) =0.4802158
##     class counts:  1010  1126
##    probabilities: 0.473 0.527 
##   left son=24 (2015 obs) right son=25 (121 obs)
##   Primary splits:
##       Hhold.fctrSKy    < 0.5 to the left,  improve=4.6065270, (0 missing)
##       .clusterid.fctr4 < 0.5 to the left,  improve=3.7374020, (0 missing)
##       .clusterid.fctr3 < 0.5 to the left,  improve=1.8085500, (0 missing)
##       Hhold.fctrSKn    < 0.5 to the right, improve=1.4825360, (0 missing)
##       Hhold.fctrPKy    < 0.5 to the left,  improve=0.8437651, (0 missing)
## 
## Node number 13: 866 observations,    complexity param=0.0023912
##   predicted class=D  expected loss=0.3972286  P(node) =0.1946942
##     class counts:   344   522
##    probabilities: 0.397 0.603 
##   left son=26 (297 obs) right son=27 (569 obs)
##   Primary splits:
##       Hhold.fctrSKn < 0.5 to the left,  improve=11.1767300, (0 missing)
##       Hhold.fctrMKn < 0.5 to the right, improve= 2.0017040, (0 missing)
##       Hhold.fctrSKy < 0.5 to the right, improve= 1.7310560, (0 missing)
##       Hhold.fctrPKy < 0.5 to the right, improve= 0.8636391, (0 missing)
##   Surrogate splits:
##       Hhold.fctrMKn < 0.5 to the right, agree=0.801, adj=0.421, (0 split)
##       Hhold.fctrSKy < 0.5 to the right, agree=0.687, adj=0.088, (0 split)
##       Hhold.fctrPKy < 0.5 to the right, agree=0.673, adj=0.047, (0 split)
## 
## Node number 24: 2015 observations,    complexity param=0.004208513
##   predicted class=D  expected loss=0.4808933  P(node) =0.4530126
##     class counts:   969  1046
##    probabilities: 0.481 0.519 
##   left son=48 (1708 obs) right son=49 (307 obs)
##   Primary splits:
##       .clusterid.fctr4 < 0.5 to the left,  improve=4.6639860, (0 missing)
##       .clusterid.fctr3 < 0.5 to the left,  improve=2.5688400, (0 missing)
##       Hhold.fctrPKy    < 0.5 to the left,  improve=0.9798796, (0 missing)
##       Hhold.fctrSKn    < 0.5 to the right, improve=0.3104652, (0 missing)
##       Hhold.fctrMKn    < 0.5 to the left,  improve=0.1030380, (0 missing)
## 
## Node number 25: 121 observations
##   predicted class=D  expected loss=0.338843  P(node) =0.02720324
##     class counts:    41    80
##    probabilities: 0.339 0.661 
## 
## Node number 26: 297 observations,    complexity param=0.0023912
##   predicted class=R  expected loss=0.4915825  P(node) =0.06677158
##     class counts:   151   146
##    probabilities: 0.508 0.492 
##   left son=52 (172 obs) right son=53 (125 obs)
##   Primary splits:
##       Hhold.fctrMKn < 0.5 to the left,  improve=0.3486101, (0 missing)
##       Hhold.fctrSKy < 0.5 to the right, improve=0.2674498, (0 missing)
##       Hhold.fctrPKy < 0.5 to the right, improve=0.1166707, (0 missing)
## 
## Node number 27: 569 observations
##   predicted class=D  expected loss=0.3391916  P(node) =0.1279227
##     class counts:   193   376
##    probabilities: 0.339 0.661 
## 
## Node number 48: 1708 observations,    complexity param=0.004208513
##   predicted class=D  expected loss=0.4953162  P(node) =0.3839928
##     class counts:   846   862
##    probabilities: 0.495 0.505 
##   left son=96 (1252 obs) right son=97 (456 obs)
##   Primary splits:
##       .clusterid.fctr3 < 0.5 to the left,  improve=4.64559000, (0 missing)
##       Hhold.fctrSKn    < 0.5 to the right, improve=0.76958710, (0 missing)
##       Hhold.fctrMKn    < 0.5 to the left,  improve=0.62013200, (0 missing)
##       Hhold.fctrPKy    < 0.5 to the right, improve=0.03091713, (0 missing)
## 
## Node number 49: 307 observations
##   predicted class=D  expected loss=0.4006515  P(node) =0.06901978
##     class counts:   123   184
##    probabilities: 0.401 0.599 
## 
## Node number 52: 172 observations
##   predicted class=R  expected loss=0.4709302  P(node) =0.03866906
##     class counts:    91    81
##    probabilities: 0.529 0.471 
## 
## Node number 53: 125 observations
##   predicted class=D  expected loss=0.48  P(node) =0.02810252
##     class counts:    60    65
##    probabilities: 0.480 0.520 
## 
## Node number 96: 1252 observations
##   predicted class=R  expected loss=0.4824281  P(node) =0.2814748
##     class counts:   648   604
##    probabilities: 0.518 0.482 
## 
## Node number 97: 456 observations
##   predicted class=D  expected loss=0.4342105  P(node) =0.102518
##     class counts:   198   258
##    probabilities: 0.434 0.566 
## 
## n= 4448 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
##  1) root 4448 2091 D (0.4700989 0.5299011)  
##    2) Hhold.fctrMKy>=0.5 1296  594 R (0.5416667 0.4583333)  
##      4) .clusterid.fctr3< 0.5 1087  459 R (0.5777369 0.4222631) *
##      5) .clusterid.fctr3>=0.5 209   74 D (0.3540670 0.6459330) *
##    3) Hhold.fctrMKy< 0.5 3152 1389 D (0.4406726 0.5593274)  
##      6) Hhold.fctrPKn< 0.5 3002 1354 D (0.4510326 0.5489674)  
##       12) .clusterid.fctr2< 0.5 2136 1010 D (0.4728464 0.5271536)  
##         24) Hhold.fctrSKy< 0.5 2015  969 D (0.4808933 0.5191067)  
##           48) .clusterid.fctr4< 0.5 1708  846 D (0.4953162 0.5046838)  
##             96) .clusterid.fctr3< 0.5 1252  604 R (0.5175719 0.4824281) *
##             97) .clusterid.fctr3>=0.5 456  198 D (0.4342105 0.5657895) *
##           49) .clusterid.fctr4>=0.5 307  123 D (0.4006515 0.5993485) *
##         25) Hhold.fctrSKy>=0.5 121   41 D (0.3388430 0.6611570) *
##       13) .clusterid.fctr2>=0.5 866  344 D (0.3972286 0.6027714)  
##         26) Hhold.fctrSKn< 0.5 297  146 R (0.5084175 0.4915825)  
##           52) Hhold.fctrMKn< 0.5 172   81 R (0.5290698 0.4709302) *
##           53) Hhold.fctrMKn>=0.5 125   60 D (0.4800000 0.5200000) *
##         27) Hhold.fctrSKn>=0.5 569  193 D (0.3391916 0.6608084) *
##      7) Hhold.fctrPKn>=0.5 150   35 D (0.2333333 0.7666667) *
## [1] "myfit_mdl: train diagnostics complete: 3.365000 secs"

##          Prediction
## Reference    R    D
##         R 2056   35
##         D 2242  115
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##     0.48808453     0.03025076     0.47329434     0.50289038     0.52990108 
## AccuracyPValue  McnemarPValue 
##     0.99999999     0.00000000

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 7.861000 secs"
##                     id                      feats max.nTuningRuns
## 1 Max.cor.Y##rcv#rpart .clusterid.fctr,Hhold.fctr               5
##   min.elapsedtime.everything min.elapsedtime.final max.AUCpROC.fit
## 1                      1.769                 0.032       0.5841957
##   max.Sens.fit max.Spec.fit max.AUCROCR.fit opt.prob.threshold.fit
## 1    0.6537542    0.5146373       0.3939105                    0.7
##   max.f.score.fit max.Accuracy.fit max.AccuracyLower.fit
## 1       0.6436062        0.5696192             0.4732943
##   max.AccuracyUpper.fit max.Kappa.fit max.AUCpROC.OOB max.Sens.OOB
## 1             0.5028904     0.1403183       0.5502714    0.5988593
##   max.Spec.OOB max.AUCROCR.OOB opt.prob.threshold.OOB max.f.score.OOB
## 1    0.5016835       0.4470465                    0.8       0.6391252
##   max.Accuracy.OOB max.AccuracyLower.OOB max.AccuracyUpper.OOB
## 1        0.4696429             0.4400805             0.4993651
##   max.Kappa.OOB max.AccuracySD.fit max.KappaSD.fit
## 1             0        0.008483058      0.01667115
## [1] "myfit_mdl: exit: 7.877000 secs"

if ((length(glbFeatsDateTime) > 0) && 
    (sum(grepl(paste(names(glbFeatsDateTime), "\\.day\\.minutes\\.poly\\.", sep = ""),
               names(glbObsAll))) > 0)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Max.cor.Y.Time.Poly"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars, 
            grep(paste(names(glbFeatsDateTime), "\\.day\\.minutes\\.poly\\.", sep = ""),
                        names(glbObsAll), value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = "Max.cor.Y.Time.Poly", 
            type = glb_model_type, trainControl.method = "repeatedcv",
            trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,            
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

if ((length(glbFeatsDateTime) > 0) && 
    (sum(grepl(paste(names(glbFeatsDateTime), "\\.last[[:digit:]]", sep = ""),
               names(glbObsAll))) > 0)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Max.cor.Y.Time.Lag"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars, 
            grep(paste(names(glbFeatsDateTime), "\\.last[[:digit:]]", sep = ""),
                        names(glbObsAll), value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Time.Lag", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,        
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

if (length(glbFeatsText) > 0) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Txt.*"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.(?!([T|P]\\.))", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.nonTP", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,                                
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.T\\.", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.onlyT", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.P\\.", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.onlyP", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,        
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

# Interactions.High.cor.Y
if (length(int_feats <- setdiff(setdiff(unique(glb_feats_df$cor.high.X), NA), 
                                subset(glb_feats_df, nzv)$id)) > 0) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Interact.High.cor.Y"), major.inc = FALSE,
                                    label.minor = "glmnet")

    ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
        id.prefix="Interact.High.cor.Y", 
        type=glb_model_type, trainControl.method="repeatedcv",
        trainControl.number=glb_rcv_n_folds, trainControl.repeats=glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method="glmnet")),
        indepVar=c(max_cor_y_x_vars, paste(max_cor_y_x_vars[1], int_feats, sep=":")),
        rsp_var=glb_rsp_var, 
        fit_df=glbObsFit, OOB_df=glbObsOOB)
}    

# Low.cor.X
fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                        paste0("fit.models_0_", "Low.cor.X"), major.inc = FALSE,
                                     label.minor = "glmnet")

##                            label step_major step_minor label_minor     bgn
## 4 fit.models_0_Max.cor.Y.rcv.*X*          1          3      glmnet 259.053
## 5         fit.models_0_Low.cor.X          1          4      glmnet 273.095
##       end elapsed
## 4 273.094  14.042
## 5      NA      NA

indepVar <- mygetIndepVar(glb_feats_df)
ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = "Low.cor.X", 
            type = glb_model_type, 
            tune.df = glbMdlTuneParams,        
            trainControl.method = "repeatedcv",
            trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = "glmnet")),
        indepVar = indepVar, rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Low.cor.X##rcv#glmnet"
## [1] "    indepVar: Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.734000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 0.775, lambda = 0.0216 on full training set
## [1] "myfit_mdl: train complete: 3.855000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = "Low.cor.X", : model's bestTune found at an extreme of
## tuneGrid for parameter: lambda

##             Length Class      Mode     
## a0            49   -none-     numeric  
## beta        1372   dgCMatrix  S4       
## df            49   -none-     numeric  
## dim            2   -none-     numeric  
## lambda        49   -none-     numeric  
## dev.ratio     49   -none-     numeric  
## nulldev        1   -none-     numeric  
## npasses        1   -none-     numeric  
## jerr           1   -none-     numeric  
## offset         1   -none-     logical  
## classnames     2   -none-     character
## call           5   -none-     call     
## nobs           1   -none-     numeric  
## lambdaOpt      1   -none-     numeric  
## xNames        28   -none-     character
## problemType    1   -none-     character
## tuneValue      2   data.frame list     
## obsLevels      2   -none-     character
## [1] "min lambda > lambdaOpt:"
##                    (Intercept)                  Hhold.fctrMKy 
##                     0.09290574                    -0.20257296 
##                  Hhold.fctrPKn Hhold.fctrPKn:.clusterid.fctr2 
##                     0.54225822                     0.18596185 
## Hhold.fctrSKn:.clusterid.fctr2 Hhold.fctrMKn:.clusterid.fctr3 
##                     0.36325477                     0.18138347 
## Hhold.fctrMKy:.clusterid.fctr3 Hhold.fctrPKy:.clusterid.fctr4 
##                     0.36291452                     0.26643652 
## [1] "max lambda < lambdaOpt:"
##                    (Intercept)                  Hhold.fctrMKy 
##                    0.087119921                   -0.214397161 
##                  Hhold.fctrPKn                  Hhold.fctrSKy 
##                    0.572535797                    0.033362924 
## Hhold.fctrPKn:.clusterid.fctr2 Hhold.fctrSKn:.clusterid.fctr2 
##                    0.231879816                    0.386886071 
## Hhold.fctrMKn:.clusterid.fctr3 Hhold.fctrMKy:.clusterid.fctr3 
##                    0.236447578                    0.409670223 
## Hhold.fctrPKy:.clusterid.fctr4 Hhold.fctrSKn:.clusterid.fctr4 
##                    0.393799216                    0.006125892 
## [1] "myfit_mdl: train diagnostics complete: 4.524000 secs"

##          Prediction
## Reference    R    D
##         R 2056   35
##         D 2242  115
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##     0.48808453     0.03025076     0.47329434     0.50289038     0.52990108 
## AccuracyPValue  McnemarPValue 
##     0.99999999     0.00000000

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 9.026000 secs"
##                      id                                        feats
## 1 Low.cor.X##rcv#glmnet Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              20                      3.109                 0.104
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.5527978    0.3003348    0.8052609       0.3971144
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                   0.65       0.6436062        0.5678975
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.4732943             0.5028904     0.1085324
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1       0.5229705    0.2395437    0.8063973       0.4464208
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                   0.75       0.6391252        0.4696429
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.4400805             0.4993651             0
##   max.AccuracySD.fit max.KappaSD.fit
## 1         0.01478065      0.03075101
## [1] "myfit_mdl: exit: 9.041000 secs"

fit.models_0_chunk_df <- 
    myadd_chunk(fit.models_0_chunk_df, "fit.models_0_end", major.inc = FALSE,
                label.minor = "teardown")

##                    label step_major step_minor label_minor     bgn     end
## 5 fit.models_0_Low.cor.X          1          4      glmnet 273.095 282.159
## 6       fit.models_0_end          1          5    teardown 282.160      NA
##   elapsed
## 5   9.064
## 6      NA

rm(ret_lst)

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc = FALSE)

##        label step_major step_minor label_minor     bgn     end elapsed
## 4 fit.models          4          0           0 245.736 282.173  36.437
## 5 fit.models          4          1           1 282.174      NA      NA

fit.models_1_chunk_df <- myadd_chunk(NULL, "fit.models_1_bgn", label.minor = "setup")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_1_bgn          1          0       setup 285.706  NA      NA

# refactor code for outliers / ensure all model runs exclude outliers in this chunk ???

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)
topindep_var <- NULL; interact_vars <- NULL;
for (mdl_id_pfx in names(glbMdlFamilies)) {
    fit.models_1_chunk_df <- 
        myadd_chunk(fit.models_1_chunk_df, paste0("fit.models_1_", mdl_id_pfx),
                    major.inc = FALSE, label.minor = "setup")

    indepVar <- NULL;

    if (grepl("\\.Interact", mdl_id_pfx)) {
        if (is.null(topindep_var) && is.null(interact_vars)) {
        #   select best glmnet model upto now
            dsp_models_df <- orderBy(model_sel_frmla <- get_model_sel_frmla(),
                                     glb_models_df)
            dsp_models_df <- subset(dsp_models_df, 
                                    grepl(".glmnet", id, fixed = TRUE))
            bst_mdl_id <- dsp_models_df$id[1]
            mdl_id_pfx <- 
                paste(c(head(unlist(strsplit(bst_mdl_id, "[.]")), -1), "Interact"),
                      collapse=".")
        #   select important features
            if (is.null(bst_featsimp_df <- 
                        myget_feats_importance(glb_models_lst[[bst_mdl_id]]))) {
                warning("Base model for RFE.Interact: ", bst_mdl_id, 
                        " has no important features")
                next
            }    
            
            topindep_ix <- 1
            while (is.null(topindep_var) && (topindep_ix <= nrow(bst_featsimp_df))) {
                topindep_var <- row.names(bst_featsimp_df)[topindep_ix]
                if (grepl(".fctr", topindep_var, fixed=TRUE))
                    topindep_var <- 
                        paste0(unlist(strsplit(topindep_var, ".fctr"))[1], ".fctr")
                if (topindep_var %in% names(glbFeatsInteractionOnly)) {
                    topindep_var <- NULL; topindep_ix <- topindep_ix + 1
                } else break
            }
            
        #   select features with importance > max(10, importance of .rnorm) & is not highest
        #       combine factor dummy features to just the factor feature
            if (length(pos_rnorm <- 
                       grep(".rnorm", row.names(bst_featsimp_df), fixed=TRUE)) > 0)
                imp_rnorm <- bst_featsimp_df[pos_rnorm, 1] else
                imp_rnorm <- NA    
            imp_cutoff <- max(10, imp_rnorm, na.rm=TRUE)
            interact_vars <- 
                tail(row.names(subset(bst_featsimp_df, 
                                      imp > imp_cutoff)), -1)
            if (length(interact_vars) > 0) {
                interact_vars <-
                    myadjustInteractionFeats(glb_feats_df, myextract_actual_feats(interact_vars))
                interact_vars <- 
                    interact_vars[!grepl(topindep_var, interact_vars, fixed=TRUE)]
            }
            ### bid0_sp only
#             interact_vars <- c(
#     "biddable", "D.ratio.sum.TfIdf.wrds.n", "D.TfIdf.sum.stem.stop.Ratio", "D.sum.TfIdf",
#     "D.TfIdf.sum.post.stop", "D.TfIdf.sum.post.stem", "D.ratio.wrds.stop.n.wrds.n", "D.chrs.uppr.n.log",
#     "D.chrs.n.log", "color.fctr"
#     # , "condition.fctr", "prdl.my.descr.fctr"
#                                 )
#            interact_vars <- setdiff(interact_vars, c("startprice.dgt2.is9", "color.fctr"))
            ###
            indepVar <- myextract_actual_feats(row.names(bst_featsimp_df))
            indepVar <- setdiff(indepVar, topindep_var)
            if (length(interact_vars) > 0) {
                indepVar <- 
                    setdiff(indepVar, myextract_actual_feats(interact_vars))
                indepVar <- c(indepVar, 
                    paste(topindep_var, setdiff(interact_vars, topindep_var), 
                          sep = "*"))
            } else indepVar <- union(indepVar, topindep_var)
        }
    }
    
    if (is.null(indepVar))
        indepVar <- glb_mdl_feats_lst[[mdl_id_pfx]]

    if (is.null(indepVar) && grepl("RFE\\.", mdl_id_pfx))
        indepVar <- myextract_actual_feats(predictors(rfe_fit_results))
    
    if (is.null(indepVar))
        indepVar <- mygetIndepVar(glb_feats_df)
    
    if ((length(indepVar) == 1) && (grepl("^%<d-%", indepVar))) {    
        indepVar <- 
            eval(parse(text = str_trim(unlist(strsplit(indepVar, "%<d-%"))[2])))
    }    

    indepVar <- myadjustInteractionFeats(glb_feats_df, indepVar)
    
    if (grepl("\\.Interact", mdl_id_pfx)) { 
        # if (method != tail(unlist(strsplit(bst_mdl_id, "[.]")), 1)) next
        if (is.null(glbMdlFamilies[[mdl_id_pfx]])) {
            if (!is.null(glbMdlFamilies[["Best.Interact"]]))
                glbMdlFamilies[[mdl_id_pfx]] <-
                    glbMdlFamilies[["Best.Interact"]]
        }
    }
    
    if (!is.null(glbObsFitOutliers[[mdl_id_pfx]])) {
        fitobs_df <- glbObsFit[!(glbObsFit[, glbFeatsId] %in%
                                         glbObsFitOutliers[[mdl_id_pfx]]), ]
        print(sprintf("Outliers removed: %d", nrow(glbObsFit) - nrow(fitobs_df)))
        print(setdiff(glbObsFit[, glbFeatsId], fitobs_df[, glbFeatsId]))
    } else fitobs_df <- glbObsFit

    if (is.null(glbMdlFamilies[[mdl_id_pfx]]))
        mdl_methods <- glbMdlMethods else
        mdl_methods <- glbMdlFamilies[[mdl_id_pfx]]    

    for (method in mdl_methods) {
        if (method %in% c("rpart", "rf")) {
            # rpart:    fubar's the tree
            # rf:       skip the scenario w/ .rnorm for speed
            indepVar <- setdiff(indepVar, c(".rnorm"))
            #mdl_id <- paste0(mdl_id_pfx, ".no.rnorm")
        } 

        fit.models_1_chunk_df <- myadd_chunk(fit.models_1_chunk_df, 
                            paste0("fit.models_1_", mdl_id_pfx), major.inc = FALSE,
                                    label.minor = method)

        ret_lst <- 
            myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = mdl_id_pfx, 
            type = glb_model_type, 
            tune.df = glbMdlTuneParams,
            trainControl.method = "repeatedcv", # or "none" if nominalWorkflow is crashing
            trainControl.number = glb_rcv_n_folds,
            trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,            
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = method)),
            indepVar = indepVar, rsp_var = glb_rsp_var, 
            fit_df = fitobs_df, OOB_df = glbObsOOB)
        
#         ntv_mdl <- glmnet(x = as.matrix(
#                               fitobs_df[, indepVar]), 
#                           y = as.factor(as.character(
#                               fitobs_df[, glb_rsp_var])),
#                           family = "multinomial")
#         bgn = 1; end = 100;
#         ntv_mdl <- glmnet(x = as.matrix(
#                               subset(fitobs_df, pop.fctr != "crypto")[bgn:end, indepVar]), 
#                           y = as.factor(as.character(
#                               subset(fitobs_df, pop.fctr != "crypto")[bgn:end, glb_rsp_var])),
#                           family = "multinomial")
    }
}

##                label step_major step_minor label_minor     bgn     end
## 1   fit.models_1_bgn          1          0       setup 285.706 285.717
## 2 fit.models_1_All.X          1          1       setup 285.718      NA
##   elapsed
## 1   0.011
## 2      NA
##                label step_major step_minor label_minor     bgn     end
## 2 fit.models_1_All.X          1          1       setup 285.718 285.727
## 3 fit.models_1_All.X          1          2      glmnet 285.728      NA
##   elapsed
## 2   0.009
## 3      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: All.X##rcv#glmnet"
## [1] "    indepVar: Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.736000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 0.775, lambda = 0.0216 on full training set
## [1] "myfit_mdl: train complete: 3.931000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = mdl_id_pfx, : model's bestTune found at an extreme of
## tuneGrid for parameter: lambda

##             Length Class      Mode     
## a0            49   -none-     numeric  
## beta        1372   dgCMatrix  S4       
## df            49   -none-     numeric  
## dim            2   -none-     numeric  
## lambda        49   -none-     numeric  
## dev.ratio     49   -none-     numeric  
## nulldev        1   -none-     numeric  
## npasses        1   -none-     numeric  
## jerr           1   -none-     numeric  
## offset         1   -none-     logical  
## classnames     2   -none-     character
## call           5   -none-     call     
## nobs           1   -none-     numeric  
## lambdaOpt      1   -none-     numeric  
## xNames        28   -none-     character
## problemType    1   -none-     character
## tuneValue      2   data.frame list     
## obsLevels      2   -none-     character
## [1] "min lambda > lambdaOpt:"
##                    (Intercept)                  Hhold.fctrMKy 
##                     0.09290574                    -0.20257296 
##                  Hhold.fctrPKn Hhold.fctrPKn:.clusterid.fctr2 
##                     0.54225822                     0.18596185 
## Hhold.fctrSKn:.clusterid.fctr2 Hhold.fctrMKn:.clusterid.fctr3 
##                     0.36325477                     0.18138347 
## Hhold.fctrMKy:.clusterid.fctr3 Hhold.fctrPKy:.clusterid.fctr4 
##                     0.36291452                     0.26643652 
## [1] "max lambda < lambdaOpt:"
##                    (Intercept)                  Hhold.fctrMKy 
##                    0.087119921                   -0.214397161 
##                  Hhold.fctrPKn                  Hhold.fctrSKy 
##                    0.572535797                    0.033362924 
## Hhold.fctrPKn:.clusterid.fctr2 Hhold.fctrSKn:.clusterid.fctr2 
##                    0.231879816                    0.386886071 
## Hhold.fctrMKn:.clusterid.fctr3 Hhold.fctrMKy:.clusterid.fctr3 
##                    0.236447578                    0.409670223 
## Hhold.fctrPKy:.clusterid.fctr4 Hhold.fctrSKn:.clusterid.fctr4 
##                    0.393799216                    0.006125892 
## [1] "myfit_mdl: train diagnostics complete: 8.623000 secs"

##          Prediction
## Reference    R    D
##         R 2056   35
##         D 2242  115
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##     0.48808453     0.03025076     0.47329434     0.50289038     0.52990108 
## AccuracyPValue  McnemarPValue 
##     0.99999999     0.00000000

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 13.928000 secs"
##                  id                                        feats
## 1 All.X##rcv#glmnet Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              20                      3.182                 0.102
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.5527978    0.3003348    0.8052609       0.3971144
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                   0.65       0.6436062        0.5678975
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.4732943             0.5028904     0.1085324
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1       0.5229705    0.2395437    0.8063973       0.4464208
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                   0.75       0.6391252        0.4696429
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.4400805             0.4993651             0
##   max.AccuracySD.fit max.KappaSD.fit
## 1         0.01478065      0.03075101
## [1] "myfit_mdl: exit: 13.942000 secs"
##                label step_major step_minor label_minor     bgn     end
## 3 fit.models_1_All.X          1          2      glmnet 285.728 299.676
## 4 fit.models_1_All.X          1          3         glm 299.676      NA
##   elapsed
## 3  13.948
## 4      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: All.X##rcv#glm"
## [1] "    indepVar: Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.704000 secs"
## + Fold1.Rep1: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold1.Rep1: parameter=none 
## + Fold2.Rep1: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold2.Rep1: parameter=none 
## + Fold3.Rep1: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold3.Rep1: parameter=none 
## + Fold1.Rep2: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold1.Rep2: parameter=none 
## + Fold2.Rep2: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold2.Rep2: parameter=none 
## + Fold3.Rep2: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold3.Rep2: parameter=none 
## + Fold1.Rep3: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold1.Rep3: parameter=none 
## + Fold2.Rep3: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold2.Rep3: parameter=none 
## + Fold3.Rep3: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold3.Rep3: parameter=none 
## Aggregating results
## Fitting final model on full training set
## [1] "myfit_mdl: train complete: 2.706000 secs"

## 
## Call:
## NULL
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.2641  -1.1477   0.8317   1.1598   1.3770  
## 
## Coefficients: (5 not defined because of singularities)
##                                  Estimate Std. Error z value Pr(>|z|)    
## (Intercept)                       0.04031    0.15868   0.254 0.799455    
## .rnorm                           -0.02397    0.03036  -0.789 0.429821    
## Hhold.fctrMKn                    -0.08949    0.19452  -0.460 0.645471    
## Hhold.fctrMKy                    -0.42554    0.17319  -2.457 0.014007 *  
## Hhold.fctrPKn                     0.95893    0.30057   3.190 0.001421 ** 
## Hhold.fctrPKy                    -0.03239    0.55763  -0.058 0.953686    
## Hhold.fctrSKn                    -0.14233    0.17446  -0.816 0.414603    
## Hhold.fctrSKy                     0.76802    0.27394   2.804 0.005054 ** 
## `Hhold.fctrN:.clusterid.fctr2`   -0.10249    0.23565  -0.435 0.663613    
## `Hhold.fctrMKn:.clusterid.fctr2`  0.12693    0.21153   0.600 0.548467    
## `Hhold.fctrMKy:.clusterid.fctr2`  0.34126    0.15086   2.262 0.023689 *  
## `Hhold.fctrPKn:.clusterid.fctr2`  0.83726    0.48039   1.743 0.081355 .  
## `Hhold.fctrPKy:.clusterid.fctr2` -0.29242    0.75991  -0.385 0.700382    
## `Hhold.fctrSKn:.clusterid.fctr2`  0.76859    0.11445   6.715 1.88e-11 ***
## `Hhold.fctrSKy:.clusterid.fctr2` -1.12282    0.45555  -2.465 0.013711 *  
## `Hhold.fctrN:.clusterid.fctr3`    0.11677    0.27957   0.418 0.676174    
## `Hhold.fctrMKn:.clusterid.fctr3`  0.92990    0.27749   3.351 0.000805 ***
## `Hhold.fctrMKy:.clusterid.fctr3`  0.98469    0.16040   6.139 8.30e-10 ***
## `Hhold.fctrPKn:.clusterid.fctr3` -0.30930    0.52867  -0.585 0.558521    
## `Hhold.fctrPKy:.clusterid.fctr3` -0.19917    0.80801  -0.246 0.805302    
## `Hhold.fctrSKn:.clusterid.fctr3`  0.26687    0.13769   1.938 0.052594 .  
## `Hhold.fctrSKy:.clusterid.fctr3` -0.58387    0.44708  -1.306 0.191572    
## `Hhold.fctrN:.clusterid.fctr4`         NA         NA      NA       NA    
## `Hhold.fctrMKn:.clusterid.fctr4`       NA         NA      NA       NA    
## `Hhold.fctrMKy:.clusterid.fctr4`       NA         NA      NA       NA    
## `Hhold.fctrPKn:.clusterid.fctr4`       NA         NA      NA       NA    
## `Hhold.fctrPKy:.clusterid.fctr4`  2.49259    1.16944   2.131 0.033052 *  
## `Hhold.fctrSKn:.clusterid.fctr4`  0.44626    0.13882   3.215 0.001306 ** 
## `Hhold.fctrSKy:.clusterid.fctr4`       NA         NA      NA       NA    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 6150.3  on 4447  degrees of freedom
## Residual deviance: 5953.4  on 4424  degrees of freedom
## AIC: 6001.4
## 
## Number of Fisher Scoring iterations: 4
## 
## [1] "myfit_mdl: train diagnostics complete: 3.924000 secs"

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

##          Prediction
## Reference    R    D
##         R 2040   51
##         D 2196  161
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##      0.4948291      0.0415139      0.4800311      0.5096339      0.5299011 
## AccuracyPValue  McnemarPValue 
##      0.9999987      0.0000000

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

##          Prediction
## Reference   R   D
##         R 526   0
##         D 593   1
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.705357e-01   1.581451e-03   4.409678e-01   5.002587e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999729e-01  1.517149e-130 
## [1] "myfit_mdl: predict complete: 11.623000 secs"
##               id                                        feats
## 1 All.X##rcv#glm Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1               1                       1.99                 0.075
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.5833366    0.6159732       0.5507       0.3857553
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                    0.7       0.6448554        0.5647478
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.4800311             0.5096339     0.1286029
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1       0.5576807    0.5665399    0.5488215       0.4324359
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                    0.9       0.6395137        0.4705357
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.4409678             0.5002587   0.001581451
##   max.AccuracySD.fit max.KappaSD.fit
## 1         0.01005124      0.01986131
## [1] "myfit_mdl: exit: 11.638000 secs"

# Check if other preProcess methods improve model performance
fit.models_1_chunk_df <- 
    myadd_chunk(fit.models_1_chunk_df, "fit.models_1_preProc", major.inc = FALSE,
                label.minor = "preProc")

##                  label step_major step_minor label_minor     bgn    end
## 4   fit.models_1_All.X          1          3         glm 299.676 311.36
## 5 fit.models_1_preProc          1          4     preProc 311.361     NA
##   elapsed
## 4  11.684
## 5      NA

require(gdata)

## Loading required package: gdata

## gdata: read.xls support for 'XLS' (Excel 97-2004) files ENABLED.

##

## gdata: read.xls support for 'XLSX' (Excel 2007+) files ENABLED.

## 
## Attaching package: 'gdata'

## The following objects are masked from 'package:dplyr':
## 
##     combine, first, last

## The following object is masked from 'package:stats':
## 
##     nobs

## The following object is masked from 'package:utils':
## 
##     object.size

mdl_id <- orderBy(get_model_sel_frmla(), glb_models_df)[1, "id"]
indepVar <- trim(unlist(strsplit(glb_models_df[glb_models_df$id == mdl_id,
                                                      "feats"], "[,]")))
method <- tail(unlist(strsplit(mdl_id, "[.]")), 1)
mdl_id_pfx <- paste0(head(unlist(strsplit(mdl_id, "[.]")), -1), collapse = ".")
if (!is.null(glbObsFitOutliers[[mdl_id_pfx]])) {
    fitobs_df <- glbObsFit[!(glbObsFit[, glbFeatsId] %in%
                                     glbObsFitOutliers[[mdl_id_pfx]]), ]
        print(sprintf("Outliers removed: %d", nrow(glbObsFit) - nrow(fitobs_df)))
        print(setdiff(glbObsFit[, glbFeatsId], fitobs_df[, glbFeatsId]))
    
} else fitobs_df <- glbObsFit

for (prePr in glb_preproc_methods) {   
    # The operations are applied in this order: 
    #   Box-Cox/Yeo-Johnson transformation, centering, scaling, range, imputation, PCA, ICA then spatial sign.
    
    ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
            id.prefix=mdl_id_pfx, 
            type=glb_model_type, tune.df=glbMdlTuneParams,
            trainControl.method="repeatedcv",
            trainControl.number=glb_rcv_n_folds,
            trainControl.repeats=glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method=method, train.preProcess=prePr)),
            indepVar=indepVar, rsp_var=glb_rsp_var, 
            fit_df=fitobs_df, OOB_df=glbObsOOB)
}            
    
    # If (All|RFE).X.glm is less accurate than Low.Cor.X.glm
    #   check NA coefficients & filter appropriate terms in indepVar
#     if (method == "glm") {
#         orig_glm <- glb_models_lst[[paste0(mdl_id, ".", model_method)]]$finalModel
#         orig_glm <- glb_models_lst[["All.X.glm"]]$finalModel; print(summary(orig_glm))
#         orig_glm <- glb_models_lst[["RFE.X.glm"]]$finalModel; print(summary(orig_glm))
#           require(car)
#           vif_orig_glm <- vif(orig_glm); print(vif_orig_glm)
#           # if vif errors out with "there are aliased coefficients in the model"
#               alias_orig_glm <- alias(orig_glm); alias_complete_orig_glm <- (alias_orig_glm$Complete > 0); alias_complete_orig_glm <- alias_complete_orig_glm[rowSums(alias_complete_orig_glm) > 0, colSums(alias_complete_orig_glm) > 0]; print(alias_complete_orig_glm)
#           print(vif_orig_glm[!is.na(vif_orig_glm) & (vif_orig_glm == Inf)])
#           print(which.max(vif_orig_glm))
#           print(sort(vif_orig_glm[vif_orig_glm >= 1.0e+03], decreasing=TRUE))
#           glbObsFit[c(1143, 3637, 3953, 4105), c("UniqueID", "Popular", "H.P.quandary", "Headline")]
#           glb_feats_df[glb_feats_df$id %in% grep("[HSA]\\.chrs.n.log", glb_feats_df$id, value=TRUE) | glb_feats_df$cor.high.X %in%    grep("[HSA]\\.chrs.n.log", glb_feats_df$id, value=TRUE), ]
#           all.equal(glbObsAll$S.chrs.uppr.n.log, glbObsAll$A.chrs.uppr.n.log)
#           cor(glbObsAll$S.T.herald, glbObsAll$S.T.tribun)
#           mydspObs(Abstract.contains="[Dd]iar", cols=("Abstract"), all=TRUE)
#           subset(glb_feats_df, cor.y.abs <= glb_feats_df[glb_feats_df$id == ".rnorm", "cor.y.abs"])
#         corxx_mtrx <- cor(data.matrix(glbObsAll[, setdiff(names(glbObsAll), myfind_chr_cols_df(glbObsAll))]), use="pairwise.complete.obs"); abs_corxx_mtrx <- abs(corxx_mtrx); diag(abs_corxx_mtrx) <- 0
#           which.max(abs_corxx_mtrx["S.T.tribun", ])
#           abs_corxx_mtrx["A.npnct08.log", "S.npnct08.log"]
#         step_glm <- step(orig_glm)
#     }
    # Since caret does not optimize rpart well
#     if (method == "rpart")
#         ret_lst <- myfit_mdl(mdl_id=paste0(mdl_id_pfx, ".cp.0"), model_method=method,
#                                 indepVar=indepVar,
#                                 model_type=glb_model_type,
#                                 rsp_var=glb_rsp_var,
#                                 fit_df=glbObsFit, OOB_df=glbObsOOB,        
#             n_cv_folds=0, tune_models_df=data.frame(parameter="cp", min=0.0, max=0.0, by=0.1))

# User specified
#   Ensure at least 2 vars in each regression; else varImp crashes
# sav_models_lst <- glb_models_lst; sav_models_df <- glb_models_df; sav_featsimp_df <- glb_featsimp_df; all.equal(sav_featsimp_df, glb_featsimp_df)
# glb_models_lst <- sav_models_lst; glb_models_df <- sav_models_df; glm_featsimp_df <- sav_featsimp_df

    # easier to exclude features
# require(gdata) # needed for trim
# mdl_id <- "";
# indepVar <- head(subset(glb_models_df, grepl("All\\.X\\.", mdl_id), select=feats)
#                         , 1)[, "feats"]
# indepVar <- trim(unlist(strsplit(indepVar, "[,]")))
# indepVar <- setdiff(indepVar, ".rnorm")

    # easier to include features
#stop(here"); sav_models_df <- glb_models_df; glb_models_df <- sav_models_df
# !_sp
# mdl_id <- "csm"; indepVar <- c(NULL
#     ,"prdline.my.fctr", "prdline.my.fctr:.clusterid.fctr"
#     ,"prdline.my.fctr*biddable"
#     #,"prdline.my.fctr*startprice.log"
#     #,"prdline.my.fctr*startprice.diff"    
#     ,"prdline.my.fctr*condition.fctr"
#     ,"prdline.my.fctr*D.terms.post.stop.n"
#     #,"prdline.my.fctr*D.terms.post.stem.n"
#     ,"prdline.my.fctr*cellular.fctr"    
# #    ,"<feat1>:<feat2>"
#                                            )
# for (method in glbMdlMethods) {
#     ret_lst <- myfit_mdl(mdl_id=mdl_id, model_method=method,
#                                 indepVar=indepVar,
#                                 model_type=glb_model_type,
#                                 rsp_var=glb_rsp_var,
#                                 fit_df=glbObsFit, OOB_df=glbObsOOB,
#                     n_cv_folds=glb_rcv_n_folds, tune_models_df=glbMdlTuneParams)
#     csm_mdl_id <- paste0(mdl_id, ".", method)
#     csm_featsimp_df <- myget_feats_importance(glb_models_lst[[paste0(mdl_id, ".",
#                                                                      method)]]);               print(head(csm_featsimp_df))
# }
###

# Ntv.1.lm <- lm(reformulate(indepVar, glb_rsp_var), glbObsTrn); print(summary(Ntv.1.lm))

#glb_models_df[, "max.Accuracy.OOB", FALSE]
#varImp(glb_models_lst[["Low.cor.X.glm"]])
#orderBy(~ -Overall, varImp(glb_models_lst[["All.X.2.glm"]])$imp)
#orderBy(~ -Overall, varImp(glb_models_lst[["All.X.3.glm"]])$imp)
#glb_feats_df[grepl("npnct28", glb_feats_df$id), ]

    # User specified bivariate models
#     indepVar_lst <- list()
#     for (feat in setdiff(names(glbObsFit), 
#                          union(glb_rsp_var, glbFeatsExclude)))
#         indepVar_lst[["feat"]] <- feat

    # User specified combinatorial models
#     indepVar_lst <- list()
#     combn_mtrx <- combn(c("<feat1_name>", "<feat2_name>", "<featn_name>"), 
#                           <num_feats_to_choose>)
#     for (combn_ix in 1:ncol(combn_mtrx))
#         #print(combn_mtrx[, combn_ix])
#         indepVar_lst[[combn_ix]] <- combn_mtrx[, combn_ix]
    
    # template for myfit_mdl
    #   rf is hard-coded in caret to recognize only Accuracy / Kappa evaluation metrics
    #       only for OOB in trainControl ?
    
#     ret_lst <- myfit_mdl_fn(mdl_id=paste0(mdl_id_pfx, ""), model_method=method,
#                             indepVar=indepVar,
#                             rsp_var=glb_rsp_var,
#                             fit_df=glbObsFit, OOB_df=glbObsOOB,
#                             n_cv_folds=glb_rcv_n_folds, tune_models_df=glbMdlTuneParams,
#                             model_loss_mtrx=glbMdlMetric_terms,
#                             model_summaryFunction=glbMdlMetricSummaryFn,
#                             model_metric=glbMdlMetricSummary,
#                             model_metric_maximize=glbMdlMetricMaximize)

# Simplify a model
# fit_df <- glbObsFit; glb_mdl <- step(<complex>_mdl)

# Non-caret models
#     rpart_area_mdl <- rpart(reformulate("Area", response=glb_rsp_var), 
#                                data=glbObsFit, #method="class", 
#                                control=rpart.control(cp=0.12),
#                            parms=list(loss=glbMdlMetric_terms))
#     print("rpart_sel_wlm_mdl"); prp(rpart_sel_wlm_mdl)
# 

print(glb_models_df)

##                                                    id
## MFO###myMFO_classfr               MFO###myMFO_classfr
## Random###myrandom_classfr   Random###myrandom_classfr
## Max.cor.Y.rcv.1X1###glmnet Max.cor.Y.rcv.1X1###glmnet
## Max.cor.Y##rcv#rpart             Max.cor.Y##rcv#rpart
## Low.cor.X##rcv#glmnet           Low.cor.X##rcv#glmnet
## All.X##rcv#glmnet                   All.X##rcv#glmnet
## All.X##rcv#glm                         All.X##rcv#glm
##                                                                   feats
## MFO###myMFO_classfr                                              .rnorm
## Random###myrandom_classfr                                        .rnorm
## Max.cor.Y.rcv.1X1###glmnet                   .clusterid.fctr,Hhold.fctr
## Max.cor.Y##rcv#rpart                         .clusterid.fctr,Hhold.fctr
## Low.cor.X##rcv#glmnet      Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glmnet          Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glm             Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
##                            max.nTuningRuns min.elapsedtime.everything
## MFO###myMFO_classfr                      0                      0.491
## Random###myrandom_classfr                0                      0.291
## Max.cor.Y.rcv.1X1###glmnet               0                      0.823
## Max.cor.Y##rcv#rpart                     5                      1.769
## Low.cor.X##rcv#glmnet                   20                      3.109
## All.X##rcv#glmnet                       20                      3.182
## All.X##rcv#glm                           1                      1.990
##                            min.elapsedtime.final max.AUCpROC.fit
## MFO###myMFO_classfr                        0.004       0.5000000
## Random###myrandom_classfr                  0.003       0.4942483
## Max.cor.Y.rcv.1X1###glmnet                 0.074       0.5754816
## Max.cor.Y##rcv#rpart                       0.032       0.5841957
## Low.cor.X##rcv#glmnet                      0.104       0.5527978
## All.X##rcv#glmnet                          0.102       0.5527978
## All.X##rcv#glm                             0.075       0.5833366
##                            max.Sens.fit max.Spec.fit max.AUCROCR.fit
## MFO###myMFO_classfr           0.0000000    1.0000000       0.5000000
## Random###myrandom_classfr     0.4619799    0.5265168       0.5073101
## Max.cor.Y.rcv.1X1###glmnet    0.5557150    0.5952482       0.4065966
## Max.cor.Y##rcv#rpart          0.6537542    0.5146373       0.3939105
## Low.cor.X##rcv#glmnet         0.3003348    0.8052609       0.3971144
## All.X##rcv#glmnet             0.3003348    0.8052609       0.3971144
## All.X##rcv#glm                0.6159732    0.5507000       0.3857553
##                            opt.prob.threshold.fit max.f.score.fit
## MFO###myMFO_classfr                          0.50       0.6395473
## Random###myrandom_classfr                    0.55       0.6395473
## Max.cor.Y.rcv.1X1###glmnet                   0.70       0.6436062
## Max.cor.Y##rcv#rpart                         0.70       0.6436062
## Low.cor.X##rcv#glmnet                        0.65       0.6436062
## All.X##rcv#glmnet                            0.65       0.6436062
## All.X##rcv#glm                               0.70       0.6448554
##                            max.Accuracy.fit max.AccuracyLower.fit
## MFO###myMFO_classfr               0.4700989             0.4553427
## Random###myrandom_classfr         0.4700989             0.4553427
## Max.cor.Y.rcv.1X1###glmnet        0.4880845             0.4732943
## Max.cor.Y##rcv#rpart              0.5696192             0.4732943
## Low.cor.X##rcv#glmnet             0.5678975             0.4732943
## All.X##rcv#glmnet                 0.5678975             0.4732943
## All.X##rcv#glm                    0.5647478             0.4800311
##                            max.AccuracyUpper.fit max.Kappa.fit
## MFO###myMFO_classfr                    0.4848945    0.00000000
## Random###myrandom_classfr              0.4848945    0.00000000
## Max.cor.Y.rcv.1X1###glmnet             0.5028904    0.03025076
## Max.cor.Y##rcv#rpart                   0.5028904    0.14031827
## Low.cor.X##rcv#glmnet                  0.5028904    0.10853242
## All.X##rcv#glmnet                      0.5028904    0.10853242
## All.X##rcv#glm                         0.5096339    0.12860295
##                            max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB
## MFO###myMFO_classfr              0.5000000    0.0000000    1.0000000
## Random###myrandom_classfr        0.5235690    0.5000000    0.5471380
## Max.cor.Y.rcv.1X1###glmnet       0.5546626    0.5285171    0.5808081
## Max.cor.Y##rcv#rpart             0.5502714    0.5988593    0.5016835
## Low.cor.X##rcv#glmnet            0.5229705    0.2395437    0.8063973
## All.X##rcv#glmnet                0.5229705    0.2395437    0.8063973
## All.X##rcv#glm                   0.5576807    0.5665399    0.5488215
##                            max.AUCROCR.OOB opt.prob.threshold.OOB
## MFO###myMFO_classfr              0.5000000                   0.50
## Random###myrandom_classfr        0.5191202                   0.55
## Max.cor.Y.rcv.1X1###glmnet       0.4525163                   0.85
## Max.cor.Y##rcv#rpart             0.4470465                   0.80
## Low.cor.X##rcv#glmnet            0.4464208                   0.75
## All.X##rcv#glmnet                0.4464208                   0.75
## All.X##rcv#glm                   0.4324359                   0.90
##                            max.f.score.OOB max.Accuracy.OOB
## MFO###myMFO_classfr              0.6391252        0.4696429
## Random###myrandom_classfr        0.6391252        0.4696429
## Max.cor.Y.rcv.1X1###glmnet       0.6391252        0.4696429
## Max.cor.Y##rcv#rpart             0.6391252        0.4696429
## Low.cor.X##rcv#glmnet            0.6391252        0.4696429
## All.X##rcv#glmnet                0.6391252        0.4696429
## All.X##rcv#glm                   0.6395137        0.4705357
##                            max.AccuracyLower.OOB max.AccuracyUpper.OOB
## MFO###myMFO_classfr                    0.4400805             0.4993651
## Random###myrandom_classfr              0.4400805             0.4993651
## Max.cor.Y.rcv.1X1###glmnet             0.4400805             0.4993651
## Max.cor.Y##rcv#rpart                   0.4400805             0.4993651
## Low.cor.X##rcv#glmnet                  0.4400805             0.4993651
## All.X##rcv#glmnet                      0.4400805             0.4993651
## All.X##rcv#glm                         0.4409678             0.5002587
##                            max.Kappa.OOB max.AccuracySD.fit
## MFO###myMFO_classfr          0.000000000                 NA
## Random###myrandom_classfr    0.000000000                 NA
## Max.cor.Y.rcv.1X1###glmnet   0.000000000                 NA
## Max.cor.Y##rcv#rpart         0.000000000        0.008483058
## Low.cor.X##rcv#glmnet        0.000000000        0.014780651
## All.X##rcv#glmnet            0.000000000        0.014780651
## All.X##rcv#glm               0.001581451        0.010051240
##                            max.KappaSD.fit
## MFO###myMFO_classfr                     NA
## Random###myrandom_classfr               NA
## Max.cor.Y.rcv.1X1###glmnet              NA
## Max.cor.Y##rcv#rpart            0.01667115
## Low.cor.X##rcv#glmnet           0.03075101
## All.X##rcv#glmnet               0.03075101
## All.X##rcv#glm                  0.01986131

rm(ret_lst)
fit.models_1_chunk_df <- 
    myadd_chunk(fit.models_1_chunk_df, "fit.models_1_end", major.inc = FALSE,
                label.minor = "teardown")

##                  label step_major step_minor label_minor     bgn     end
## 5 fit.models_1_preProc          1          4     preProc 311.361 312.879
## 6     fit.models_1_end          1          5    teardown 312.879      NA
##   elapsed
## 5   1.518
## 6      NA

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc = FALSE)

##        label step_major step_minor label_minor     bgn     end elapsed
## 5 fit.models          4          1           1 282.174 312.889  30.716
## 6 fit.models          4          2           2 312.890      NA      NA

fit.models_2_chunk_df <- 
    myadd_chunk(NULL, "fit.models_2_bgn", label.minor = "setup")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_2_bgn          1          0       setup 316.868  NA      NA

plt_models_df <- glb_models_df[, -grep("SD|Upper|Lower", names(glb_models_df))]
for (var in grep("^min.", names(plt_models_df), value=TRUE)) {
    plt_models_df[, sub("min.", "inv.", var)] <- 
        #ifelse(all(is.na(tmp <- plt_models_df[, var])), NA, 1.0 / tmp)
        1.0 / plt_models_df[, var]
    plt_models_df <- plt_models_df[ , -grep(var, names(plt_models_df))]
}
print(plt_models_df)

##                                                    id
## MFO###myMFO_classfr               MFO###myMFO_classfr
## Random###myrandom_classfr   Random###myrandom_classfr
## Max.cor.Y.rcv.1X1###glmnet Max.cor.Y.rcv.1X1###glmnet
## Max.cor.Y##rcv#rpart             Max.cor.Y##rcv#rpart
## Low.cor.X##rcv#glmnet           Low.cor.X##rcv#glmnet
## All.X##rcv#glmnet                   All.X##rcv#glmnet
## All.X##rcv#glm                         All.X##rcv#glm
##                                                                   feats
## MFO###myMFO_classfr                                              .rnorm
## Random###myrandom_classfr                                        .rnorm
## Max.cor.Y.rcv.1X1###glmnet                   .clusterid.fctr,Hhold.fctr
## Max.cor.Y##rcv#rpart                         .clusterid.fctr,Hhold.fctr
## Low.cor.X##rcv#glmnet      Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glmnet          Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glm             Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
##                            max.nTuningRuns max.AUCpROC.fit max.Sens.fit
## MFO###myMFO_classfr                      0       0.5000000    0.0000000
## Random###myrandom_classfr                0       0.4942483    0.4619799
## Max.cor.Y.rcv.1X1###glmnet               0       0.5754816    0.5557150
## Max.cor.Y##rcv#rpart                     5       0.5841957    0.6537542
## Low.cor.X##rcv#glmnet                   20       0.5527978    0.3003348
## All.X##rcv#glmnet                       20       0.5527978    0.3003348
## All.X##rcv#glm                           1       0.5833366    0.6159732
##                            max.Spec.fit max.AUCROCR.fit
## MFO###myMFO_classfr           1.0000000       0.5000000
## Random###myrandom_classfr     0.5265168       0.5073101
## Max.cor.Y.rcv.1X1###glmnet    0.5952482       0.4065966
## Max.cor.Y##rcv#rpart          0.5146373       0.3939105
## Low.cor.X##rcv#glmnet         0.8052609       0.3971144
## All.X##rcv#glmnet             0.8052609       0.3971144
## All.X##rcv#glm                0.5507000       0.3857553
##                            opt.prob.threshold.fit max.f.score.fit
## MFO###myMFO_classfr                          0.50       0.6395473
## Random###myrandom_classfr                    0.55       0.6395473
## Max.cor.Y.rcv.1X1###glmnet                   0.70       0.6436062
## Max.cor.Y##rcv#rpart                         0.70       0.6436062
## Low.cor.X##rcv#glmnet                        0.65       0.6436062
## All.X##rcv#glmnet                            0.65       0.6436062
## All.X##rcv#glm                               0.70       0.6448554
##                            max.Accuracy.fit max.Kappa.fit max.AUCpROC.OOB
## MFO###myMFO_classfr               0.4700989    0.00000000       0.5000000
## Random###myrandom_classfr         0.4700989    0.00000000       0.5235690
## Max.cor.Y.rcv.1X1###glmnet        0.4880845    0.03025076       0.5546626
## Max.cor.Y##rcv#rpart              0.5696192    0.14031827       0.5502714
## Low.cor.X##rcv#glmnet             0.5678975    0.10853242       0.5229705
## All.X##rcv#glmnet                 0.5678975    0.10853242       0.5229705
## All.X##rcv#glm                    0.5647478    0.12860295       0.5576807
##                            max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## MFO###myMFO_classfr           0.0000000    1.0000000       0.5000000
## Random###myrandom_classfr     0.5000000    0.5471380       0.5191202
## Max.cor.Y.rcv.1X1###glmnet    0.5285171    0.5808081       0.4525163
## Max.cor.Y##rcv#rpart          0.5988593    0.5016835       0.4470465
## Low.cor.X##rcv#glmnet         0.2395437    0.8063973       0.4464208
## All.X##rcv#glmnet             0.2395437    0.8063973       0.4464208
## All.X##rcv#glm                0.5665399    0.5488215       0.4324359
##                            opt.prob.threshold.OOB max.f.score.OOB
## MFO###myMFO_classfr                          0.50       0.6391252
## Random###myrandom_classfr                    0.55       0.6391252
## Max.cor.Y.rcv.1X1###glmnet                   0.85       0.6391252
## Max.cor.Y##rcv#rpart                         0.80       0.6391252
## Low.cor.X##rcv#glmnet                        0.75       0.6391252
## All.X##rcv#glmnet                            0.75       0.6391252
## All.X##rcv#glm                               0.90       0.6395137
##                            max.Accuracy.OOB max.Kappa.OOB
## MFO###myMFO_classfr               0.4696429   0.000000000
## Random###myrandom_classfr         0.4696429   0.000000000
## Max.cor.Y.rcv.1X1###glmnet        0.4696429   0.000000000
## Max.cor.Y##rcv#rpart              0.4696429   0.000000000
## Low.cor.X##rcv#glmnet             0.4696429   0.000000000
## All.X##rcv#glmnet                 0.4696429   0.000000000
## All.X##rcv#glm                    0.4705357   0.001581451
##                            inv.elapsedtime.everything
## MFO###myMFO_classfr                         2.0366599
## Random###myrandom_classfr                   3.4364261
## Max.cor.Y.rcv.1X1###glmnet                  1.2150668
## Max.cor.Y##rcv#rpart                        0.5652911
## Low.cor.X##rcv#glmnet                       0.3216468
## All.X##rcv#glmnet                           0.3142678
## All.X##rcv#glm                              0.5025126
##                            inv.elapsedtime.final
## MFO###myMFO_classfr                   250.000000
## Random###myrandom_classfr             333.333333
## Max.cor.Y.rcv.1X1###glmnet             13.513514
## Max.cor.Y##rcv#rpart                   31.250000
## Low.cor.X##rcv#glmnet                   9.615385
## All.X##rcv#glmnet                       9.803922
## All.X##rcv#glm                         13.333333

# print(myplot_radar(radar_inp_df=plt_models_df))
# print(myplot_radar(radar_inp_df=subset(plt_models_df, 
#         !(mdl_id %in% grep("random|MFO", plt_models_df$id, value=TRUE)))))

# Compute CI for <metric>SD
glb_models_df <- mutate(glb_models_df, 
                max.df = ifelse(max.nTuningRuns > 1, max.nTuningRuns - 1, NA),
                min.sd2ci.scaler = ifelse(is.na(max.df), NA, qt(0.975, max.df)))
for (var in grep("SD", names(glb_models_df), value=TRUE)) {
    # Does CI alredy exist ?
    var_components <- unlist(strsplit(var, "SD"))
    varActul <- paste0(var_components[1],          var_components[2])
    varUpper <- paste0(var_components[1], "Upper", var_components[2])
    varLower <- paste0(var_components[1], "Lower", var_components[2])
    if (varUpper %in% names(glb_models_df)) {
        warning(varUpper, " already exists in glb_models_df")
        # Assuming Lower also exists
        next
    }    
    print(sprintf("var:%s", var))
    # CI is dependent on sample size in t distribution; df=n-1
    glb_models_df[, varUpper] <- glb_models_df[, varActul] + 
        glb_models_df[, "min.sd2ci.scaler"] * glb_models_df[, var]
    glb_models_df[, varLower] <- glb_models_df[, varActul] - 
        glb_models_df[, "min.sd2ci.scaler"] * glb_models_df[, var]
}

## Warning: max.AccuracyUpper.fit already exists in glb_models_df

## [1] "var:max.KappaSD.fit"

# Plot metrics with CI
plt_models_df <- glb_models_df[, "id", FALSE]
pltCI_models_df <- glb_models_df[, "id", FALSE]
for (var in grep("Upper", names(glb_models_df), value=TRUE)) {
    var_components <- unlist(strsplit(var, "Upper"))
    col_name <- unlist(paste(var_components, collapse=""))
    plt_models_df[, col_name] <- glb_models_df[, col_name]
    for (name in paste0(var_components[1], c("Upper", "Lower"), var_components[2]))
        pltCI_models_df[, name] <- glb_models_df[, name]
}

build_statsCI_data <- function(plt_models_df) {
    mltd_models_df <- melt(plt_models_df, id.vars="id")
    mltd_models_df$data <- sapply(1:nrow(mltd_models_df), 
        function(row_ix) tail(unlist(strsplit(as.character(
            mltd_models_df[row_ix, "variable"]), "[.]")), 1))
    mltd_models_df$label <- sapply(1:nrow(mltd_models_df), 
        function(row_ix) head(unlist(strsplit(as.character(
            mltd_models_df[row_ix, "variable"]), 
            paste0(".", mltd_models_df[row_ix, "data"]))), 1))
    #print(mltd_models_df)
    
    return(mltd_models_df)
}
mltd_models_df <- build_statsCI_data(plt_models_df)

mltdCI_models_df <- melt(pltCI_models_df, id.vars="id")
for (row_ix in 1:nrow(mltdCI_models_df)) {
    for (type in c("Upper", "Lower")) {
        if (length(var_components <- unlist(strsplit(
                as.character(mltdCI_models_df[row_ix, "variable"]), type))) > 1) {
            #print(sprintf("row_ix:%d; type:%s; ", row_ix, type))
            mltdCI_models_df[row_ix, "label"] <- var_components[1]
            mltdCI_models_df[row_ix, "data"] <- 
                unlist(strsplit(var_components[2], "[.]"))[2]
            mltdCI_models_df[row_ix, "type"] <- type
            break
        }
    }    
}
wideCI_models_df <- reshape(subset(mltdCI_models_df, select=-variable), 
                            timevar="type", 
        idvar=setdiff(names(mltdCI_models_df), c("type", "value", "variable")), 
                            direction="wide")
#print(wideCI_models_df)
mrgdCI_models_df <- merge(wideCI_models_df, mltd_models_df, all.x=TRUE)
#print(mrgdCI_models_df)

# Merge stats back in if CIs don't exist
goback_vars <- c()
for (var in unique(mltd_models_df$label)) {
    for (type in unique(mltd_models_df$data)) {
        var_type <- paste0(var, ".", type)
        # if this data is already present, next
        if (var_type %in% unique(paste(mltd_models_df$label, mltd_models_df$data,
                                       sep=".")))
            next
        #print(sprintf("var_type:%s", var_type))
        goback_vars <- c(goback_vars, var_type)
    }
}

if (length(goback_vars) > 0) {
    mltd_goback_df <- build_statsCI_data(glb_models_df[, c("id", goback_vars)])
    mltd_models_df <- rbind(mltd_models_df, mltd_goback_df)
}

# mltd_models_df <- merge(mltd_models_df, glb_models_df[, c("id", "model_method")], 
#                         all.x=TRUE)

png(paste0(glbOut$pfx, "models_bar.png"), width=480*3, height=480*2)
#print(gp <- myplot_bar(mltd_models_df, "id", "value", colorcol_name="model_method") + 
print(gp <- myplot_bar(df=mltd_models_df, xcol_name="id", ycol_names="value") + 
        geom_errorbar(data=mrgdCI_models_df, 
            mapping=aes(x=mdl_id, ymax=value.Upper, ymin=value.Lower), width=0.5) + 
          facet_grid(label ~ data, scales="free") + 
          theme(axis.text.x = element_text(angle = 90,vjust = 0.5)))

## Warning: Removed 4 rows containing missing values (geom_errorbar).

dev.off()

## quartz_off_screen 
##                 2

print(gp)

## Warning: Removed 4 rows containing missing values (geom_errorbar).

dsp_models_cols <- c("id", 
                    glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                    grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
# if (glb_is_classification && glb_is_binomial) 
#     dsp_models_cols <- c(dsp_models_cols, "opt.prob.threshold.OOB")
print(dsp_models_df <- orderBy(get_model_sel_frmla(), glb_models_df)[, dsp_models_cols])

##                           id max.Accuracy.OOB max.AUCROCR.OOB
## 7             All.X##rcv#glm        0.4705357       0.4324359
## 2  Random###myrandom_classfr        0.4696429       0.5191202
## 1        MFO###myMFO_classfr        0.4696429       0.5000000
## 3 Max.cor.Y.rcv.1X1###glmnet        0.4696429       0.4525163
## 4       Max.cor.Y##rcv#rpart        0.4696429       0.4470465
## 5      Low.cor.X##rcv#glmnet        0.4696429       0.4464208
## 6          All.X##rcv#glmnet        0.4696429       0.4464208
##   max.AUCpROC.OOB max.Accuracy.fit opt.prob.threshold.fit
## 7       0.5576807        0.5647478                   0.70
## 2       0.5235690        0.4700989                   0.55
## 1       0.5000000        0.4700989                   0.50
## 3       0.5546626        0.4880845                   0.70
## 4       0.5502714        0.5696192                   0.70
## 5       0.5229705        0.5678975                   0.65
## 6       0.5229705        0.5678975                   0.65
##   opt.prob.threshold.OOB
## 7                   0.90
## 2                   0.55
## 1                   0.50
## 3                   0.85
## 4                   0.80
## 5                   0.75
## 6                   0.75

# print(myplot_radar(radar_inp_df = dsp_models_df))
print("Metrics used for model selection:"); print(get_model_sel_frmla())

## [1] "Metrics used for model selection:"

## ~-max.Accuracy.OOB - max.AUCROCR.OOB - max.AUCpROC.OOB - max.Accuracy.fit - 
##     opt.prob.threshold.OOB
## <environment: 0x7fcdf18ee3b8>

print(sprintf("Best model id: %s", dsp_models_df[1, "id"]))

## [1] "Best model id: All.X##rcv#glm"

glb_get_predictions <- function(df, mdl_id, rsp_var, prob_threshold_def=NULL, verbose=FALSE) {
    mdl <- glb_models_lst[[mdl_id]]
    
    clmnNames <- mygetPredictIds(rsp_var, mdl_id)
    predct_var_name <- clmnNames$value        
    predct_prob_var_name <- clmnNames$prob
    predct_accurate_var_name <- clmnNames$is.acc
    predct_error_var_name <- clmnNames$err
    predct_erabs_var_name <- clmnNames$err.abs

    if (glb_is_regression) {
        df[, predct_var_name] <- predict(mdl, newdata=df, type="raw")
        if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_var_name) + 
                  facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="glm"))

        df[, predct_error_var_name] <- df[, predct_var_name] - df[, glb_rsp_var]
        if (verbose) print(myplot_scatter(df, predct_var_name, predct_error_var_name) + 
                  #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="auto"))
        if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_error_var_name) + 
                  #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="glm"))
        
        df[, predct_erabs_var_name] <- abs(df[, predct_error_var_name])
        if (verbose) print(head(orderBy(reformulate(c("-", predct_erabs_var_name)), df)))
        
        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])
    }

    if (glb_is_classification && glb_is_binomial) {
        prob_threshold <- glb_models_df[glb_models_df$id == mdl_id, 
                                        "opt.prob.threshold.OOB"]
        if (is.null(prob_threshold) || is.na(prob_threshold)) {
            warning("Using default probability threshold: ", prob_threshold_def)
            if (is.null(prob_threshold <- prob_threshold_def))
                stop("Default probability threshold is NULL")
        }
        
        df[, predct_prob_var_name] <- predict(mdl, newdata = df, type = "prob")[, 2]
        df[, predct_var_name] <- 
                factor(levels(df[, glb_rsp_var])[
                    (df[, predct_prob_var_name] >=
                        prob_threshold) * 1 + 1], levels(df[, glb_rsp_var]))
    
#         if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_var_name) + 
#                   facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="glm"))

        df[, predct_error_var_name] <- df[, predct_var_name] != df[, glb_rsp_var]
#         if (verbose) print(myplot_scatter(df, predct_var_name, predct_error_var_name) + 
#                   #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="auto"))
#         if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_error_var_name) + 
#                   #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="glm"))
        
        # if prediction is a TP (true +ve), measure distance from 1.0
        tp <- which((df[, predct_var_name] == df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[2]))
        df[tp, predct_erabs_var_name] <- abs(1 - df[tp, predct_prob_var_name])
        #rowIx <- which.max(df[tp, predct_erabs_var_name]); df[tp, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a TN (true -ve), measure distance from 0.0
        tn <- which((df[, predct_var_name] == df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[1]))
        df[tn, predct_erabs_var_name] <- abs(0 - df[tn, predct_prob_var_name])
        #rowIx <- which.max(df[tn, predct_erabs_var_name]); df[tn, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a FP (flse +ve), measure distance from 0.0
        fp <- which((df[, predct_var_name] != df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[2]))
        df[fp, predct_erabs_var_name] <- abs(0 - df[fp, predct_prob_var_name])
        #rowIx <- which.max(df[fp, predct_erabs_var_name]); df[fp, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a FN (flse -ve), measure distance from 1.0
        fn <- which((df[, predct_var_name] != df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[1]))
        df[fn, predct_erabs_var_name] <- abs(1 - df[fn, predct_prob_var_name])
        #rowIx <- which.max(df[fn, predct_erabs_var_name]); df[fn, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]

        
        if (verbose) print(head(orderBy(reformulate(c("-", predct_erabs_var_name)), df)))
        
        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])
    }    
    
    if (glb_is_classification && !glb_is_binomial) {
        df[, predct_var_name] <- predict(mdl, newdata = df, type = "raw")
        probCls <- predict(mdl, newdata = df, type = "prob")        
        df[, predct_prob_var_name] <- NA
        for (cls in names(probCls)) {
            mask <- (df[, predct_var_name] == cls)
            df[mask, predct_prob_var_name] <- probCls[mask, cls]
        }    
        if (verbose) print(myplot_histogram(df, predct_prob_var_name, 
                                            fill_col_name = predct_var_name))
        if (verbose) print(myplot_histogram(df, predct_prob_var_name, 
                                            facet_frmla = paste0("~", glb_rsp_var)))
        
        df[, predct_error_var_name] <- df[, predct_var_name] != df[, glb_rsp_var]
        
        # if prediction is erroneous, measure predicted class prob from actual class prob
        df[, predct_erabs_var_name] <- 0
        for (cls in names(probCls)) {
            mask <- (df[, glb_rsp_var] == cls) & (df[, predct_error_var_name])
            df[mask, predct_erabs_var_name] <- probCls[mask, cls]
        }    

        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])        
    }

    return(df)
}    

#stop(here"); glb2Sav(); glbObsAll <- savObsAll; glbObsTrn <- savObsTrn; glbObsFit <- savObsFit; glbObsOOB <- savObsOOB; sav_models_df <- glb_models_df; glb_models_df <- sav_models_df; glb_featsimp_df <- sav_featsimp_df    

myget_category_stats <- function(obs_df, mdl_id, label) {
    require(dplyr)
    require(lazyeval)
    
    predct_var_name <- mygetPredictIds(glb_rsp_var, mdl_id)$value        
    predct_error_var_name <- mygetPredictIds(glb_rsp_var, mdl_id)$err.abs
    
    if (!predct_var_name %in% names(obs_df))
        obs_df <- glb_get_predictions(obs_df, mdl_id, glb_rsp_var)
    
    tmp_obs_df <- obs_df[, c(glbFeatsCategory, glb_rsp_var, 
                             predct_var_name, predct_error_var_name)]
#     tmp_obs_df <- obs_df %>%
#         dplyr::select_(glbFeatsCategory, glb_rsp_var, predct_var_name, predct_error_var_name) 
    #dplyr::rename(startprice.log10.predict.RFE.X.glmnet.err=error_abs_OOB)
    names(tmp_obs_df)[length(names(tmp_obs_df))] <- paste0("err.abs.", label)
    
    ret_ctgry_df <- tmp_obs_df %>%
        dplyr::group_by_(glbFeatsCategory) %>%
        dplyr::summarise_(#interp(~sum(abs(var)), var=as.name(glb_rsp_var)), 
            interp(~sum(var), var=as.name(paste0("err.abs.", label))), 
            interp(~mean(var), var=as.name(paste0("err.abs.", label))),
            interp(~n()))
    names(ret_ctgry_df) <- c(glbFeatsCategory, 
                             #paste0(glb_rsp_var, ".abs.", label, ".sum"),
                             paste0("err.abs.", label, ".sum"),                             
                             paste0("err.abs.", label, ".mean"), 
                             paste0(".n.", label))
    ret_ctgry_df <- dplyr::ungroup(ret_ctgry_df)
    #colSums(ret_ctgry_df[, -grep(glbFeatsCategory, names(ret_ctgry_df))])
    
    return(ret_ctgry_df)    
}
#print(colSums((ctgry_df <- myget_category_stats(obs_df=glbObsFit, mdl_id="", label="fit"))[, -grep(glbFeatsCategory, names(ctgry_df))]))

if (!is.null(glb_mdl_ensemble)) {
    fit.models_2_chunk_df <- myadd_chunk(fit.models_2_chunk_df, 
                            paste0("fit.models_2_", mdl_id_pfx), major.inc = TRUE, 
                                                label.minor = "ensemble")
    
    mdl_id_pfx <- "Ensemble"

    if (#(glb_is_regression) | 
        ((glb_is_classification) & (!glb_is_binomial)))
        stop("Ensemble models not implemented yet for multinomial classification")
    
    mygetEnsembleAutoMdlIds <- function() {
        tmp_models_df <- orderBy(get_model_sel_frmla(), glb_models_df)
        row.names(tmp_models_df) <- tmp_models_df$id
        mdl_threshold_pos <- 
            min(which(grepl("MFO|Random|Baseline", tmp_models_df$id))) - 1
        mdlIds <- tmp_models_df$id[1:mdl_threshold_pos]
        return(mdlIds[!grepl("Ensemble", mdlIds)])
    }
    
    if (glb_mdl_ensemble == "auto") {
        glb_mdl_ensemble <- mygetEnsembleAutoMdlIds()
        mdl_id_pfx <- paste0(mdl_id_pfx, ".auto")        
    } else if (grepl("^%<d-%", glb_mdl_ensemble)) {
        glb_mdl_ensemble <- eval(parse(text =
                        str_trim(unlist(strsplit(glb_mdl_ensemble, "%<d-%"))[2])))
    }
    
    for (mdl_id in glb_mdl_ensemble) {
        if (!(mdl_id %in% names(glb_models_lst))) {
            warning("Model ", mdl_id, " in glb_model_ensemble not found !")
            next
        }
        glbObsFit <- glb_get_predictions(df = glbObsFit, mdl_id, glb_rsp_var)
        glbObsOOB <- glb_get_predictions(df = glbObsOOB, mdl_id, glb_rsp_var)
    }
    
#mdl_id_pfx <- "Ensemble.RFE"; mdlId <- paste0(mdl_id_pfx, ".glmnet")
#glb_mdl_ensemble <- gsub(mygetPredictIds$value, "", grep("RFE\\.X\\.(?!Interact)", row.names(glb_featsimp_df), perl = TRUE, value = TRUE), fixed = TRUE)
#varImp(glb_models_lst[[mdlId]])
    
#cor_df <- data.frame(cor=cor(glbObsFit[, glb_rsp_var], glbObsFit[, paste(mygetPredictIds$value, glb_mdl_ensemble)], use="pairwise.complete.obs"))
#glbObsFit <- glb_get_predictions(df=glbObsFit, "Ensemble.glmnet", glb_rsp_var);print(colSums((ctgry_df <- myget_category_stats(obs_df=glbObsFit, mdl_id="Ensemble.glmnet", label="fit"))[, -grep(glbFeatsCategory, names(ctgry_df))]))
    
    ### bid0_sp
    #  Better than MFO; models.n=28; min.RMSE.fit=0.0521233; err.abs.fit.sum=7.3631895
    #  old: Top x from auto; models.n= 5; min.RMSE.fit=0.06311047; err.abs.fit.sum=9.5937080
    #  RFE only ;       models.n=16; min.RMSE.fit=0.05148588; err.abs.fit.sum=7.2875091
    #  RFE subset only ;models.n= 5; min.RMSE.fit=0.06040702; err.abs.fit.sum=9.059088
    #  RFE subset only ;models.n= 9; min.RMSE.fit=0.05933167; err.abs.fit.sum=8.7421288
    #  RFE subset only ;models.n=15; min.RMSE.fit=0.0584607; err.abs.fit.sum=8.5902066
    #  RFE subset only ;models.n=17; min.RMSE.fit=0.05496899; err.abs.fit.sum=8.0170431
    #  RFE subset only ;models.n=18; min.RMSE.fit=0.05441577; err.abs.fit.sum=7.837223
    #  RFE subset only ;models.n=16; min.RMSE.fit=0.05441577; err.abs.fit.sum=7.837223
    ### bid0_sp
    ### bid1_sp
    # "auto"; err.abs.fit.sum=76.699774; min.RMSE.fit=0.2186429
    # "RFE.X.*"; err.abs.fit.sum=; min.RMSE.fit=0.221114
    ### bid1_sp

    indepVar <- paste(mygetPredictIds(glb_rsp_var)$value, glb_mdl_ensemble, sep = "")
    if (glb_is_classification)
        indepVar <- paste(indepVar, ".prob", sep = "")
    # Some models in glb_mdl_ensemble might not be fitted e.g. RFE.X.Interact
    indepVar <- intersect(indepVar, names(glbObsFit))
    
#     indepVar <- grep(mygetPredictIds(glb_rsp_var)$value, names(glbObsFit), fixed=TRUE, value=TRUE)
#     if (glb_is_regression)
#         indepVar <- indepVar[!grepl("(err\\.abs|accurate)$", indepVar)]
#     if (glb_is_classification && glb_is_binomial)
#         indepVar <- grep("prob$", indepVar, value=TRUE) else
#         indepVar <- indepVar[!grepl("err$", indepVar)]

    #rfe_fit_ens_results <- myrun_rfe(glbObsFit, indepVar)
    
    for (method in c("glm", "glmnet")) {
        for (trainControlMethod in 
             c("boot", "boot632", "cv", "repeatedcv"
               #, "LOOCV" # tuneLength * nrow(fitDF)
               , "LGOCV", "adaptive_cv"
               #, "adaptive_boot"  #error: adaptive$min should be less than 3 
               #, "adaptive_LGOCV" #error: adaptive$min should be less than 3 
               )) {
            #sav_models_df <- glb_models_df; all.equal(sav_models_df, glb_models_df)
            #glb_models_df <- sav_models_df; print(glb_models_df$id)
                
            if ((method == "glm") && (trainControlMethod != "repeatedcv"))
                # glm used only to identify outliers
                next
            
            ret_lst <- myfit_mdl(
                mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                    id.prefix = paste0(mdl_id_pfx, ".", trainControlMethod), 
                    type = glb_model_type, tune.df = NULL,
                    trainControl.method = trainControlMethod,
                    trainControl.number = glb_rcv_n_folds,
                    trainControl.repeats = glb_rcv_n_repeats,
                    trainControl.classProbs = glb_is_classification,
                    trainControl.summaryFunction = glbMdlMetricSummaryFn,
                    train.metric = glbMdlMetricSummary, 
                    train.maximize = glbMdlMetricMaximize,    
                    train.method = method)),
                indepVar = indepVar, rsp_var = glb_rsp_var, 
                fit_df = glbObsFit, OOB_df = glbObsOOB)
        }
    }
    dsp_models_df <- get_dsp_models_df()
}

if (is.null(glbMdlSelId)) 
    glbMdlSelId <- dsp_models_df[1, "id"] else 
    print(sprintf("User specified selection: %s", glbMdlSelId))

## [1] "User specified selection: All.X##rcv#glmnet"

myprint_mdl(glb_sel_mdl <- glb_models_lst[[glbMdlSelId]])

##             Length Class      Mode     
## a0            49   -none-     numeric  
## beta        1372   dgCMatrix  S4       
## df            49   -none-     numeric  
## dim            2   -none-     numeric  
## lambda        49   -none-     numeric  
## dev.ratio     49   -none-     numeric  
## nulldev        1   -none-     numeric  
## npasses        1   -none-     numeric  
## jerr           1   -none-     numeric  
## offset         1   -none-     logical  
## classnames     2   -none-     character
## call           5   -none-     call     
## nobs           1   -none-     numeric  
## lambdaOpt      1   -none-     numeric  
## xNames        28   -none-     character
## problemType    1   -none-     character
## tuneValue      2   data.frame list     
## obsLevels      2   -none-     character
## [1] "min lambda > lambdaOpt:"
##                    (Intercept)                  Hhold.fctrMKy 
##                     0.09290574                    -0.20257296 
##                  Hhold.fctrPKn Hhold.fctrPKn:.clusterid.fctr2 
##                     0.54225822                     0.18596185 
## Hhold.fctrSKn:.clusterid.fctr2 Hhold.fctrMKn:.clusterid.fctr3 
##                     0.36325477                     0.18138347 
## Hhold.fctrMKy:.clusterid.fctr3 Hhold.fctrPKy:.clusterid.fctr4 
##                     0.36291452                     0.26643652 
## [1] "max lambda < lambdaOpt:"
##                    (Intercept)                  Hhold.fctrMKy 
##                    0.087119921                   -0.214397161 
##                  Hhold.fctrPKn                  Hhold.fctrSKy 
##                    0.572535797                    0.033362924 
## Hhold.fctrPKn:.clusterid.fctr2 Hhold.fctrSKn:.clusterid.fctr2 
##                    0.231879816                    0.386886071 
## Hhold.fctrMKn:.clusterid.fctr3 Hhold.fctrMKy:.clusterid.fctr3 
##                    0.236447578                    0.409670223 
## Hhold.fctrPKy:.clusterid.fctr4 Hhold.fctrSKn:.clusterid.fctr4 
##                    0.393799216                    0.006125892

## [1] TRUE

# From here to save(), this should all be in one function
#   these are executed in the same seq twice more:
#       fit.data.training & predict.data.new chunks
print(sprintf("%s fit prediction diagnostics:", glbMdlSelId))

## [1] "All.X##rcv#glmnet fit prediction diagnostics:"

glbObsFit <- glb_get_predictions(df = glbObsFit, mdl_id = glbMdlSelId, 
                                 rsp_var = glb_rsp_var)
print(sprintf("%s OOB prediction diagnostics:", glbMdlSelId))

## [1] "All.X##rcv#glmnet OOB prediction diagnostics:"

glbObsOOB <- glb_get_predictions(df = glbObsOOB, mdl_id = glbMdlSelId, 
                                     rsp_var = glb_rsp_var)

print(glb_featsimp_df <- myget_feats_importance(mdl = glb_sel_mdl, featsimp_df = NULL))

##                                All.X..rcv.glmnet.imp         imp
## Hhold.fctrPKn                            100.0000000 100.0000000
## Hhold.fctrMKy:.clusterid.fctr3            70.6541605  70.6541605
## Hhold.fctrSKn:.clusterid.fctr2            67.4604337  67.4604337
## Hhold.fctrPKy:.clusterid.fctr4            64.9623500  64.9623500
## Hhold.fctrMKn:.clusterid.fctr3            39.7725789  39.7725789
## Hhold.fctrPKn:.clusterid.fctr2            39.2939854  39.2939854
## Hhold.fctrMKy                             37.4295156  37.4295156
## Hhold.fctrSKy                              4.6944480   4.6944480
## Hhold.fctrSKn:.clusterid.fctr4             0.8619652   0.8619652
## .rnorm                                     0.0000000   0.0000000
## Hhold.fctrMKn                              0.0000000   0.0000000
## Hhold.fctrPKy                              0.0000000   0.0000000
## Hhold.fctrSKn                              0.0000000   0.0000000
## Hhold.fctrN:.clusterid.fctr2               0.0000000   0.0000000
## Hhold.fctrMKn:.clusterid.fctr2             0.0000000   0.0000000
## Hhold.fctrMKy:.clusterid.fctr2             0.0000000   0.0000000
## Hhold.fctrPKy:.clusterid.fctr2             0.0000000   0.0000000
## Hhold.fctrSKy:.clusterid.fctr2             0.0000000   0.0000000
## Hhold.fctrN:.clusterid.fctr3               0.0000000   0.0000000
## Hhold.fctrPKn:.clusterid.fctr3             0.0000000   0.0000000
## Hhold.fctrPKy:.clusterid.fctr3             0.0000000   0.0000000
## Hhold.fctrSKn:.clusterid.fctr3             0.0000000   0.0000000
## Hhold.fctrSKy:.clusterid.fctr3             0.0000000   0.0000000
## Hhold.fctrN:.clusterid.fctr4               0.0000000   0.0000000
## Hhold.fctrMKn:.clusterid.fctr4             0.0000000   0.0000000
## Hhold.fctrMKy:.clusterid.fctr4             0.0000000   0.0000000
## Hhold.fctrPKn:.clusterid.fctr4             0.0000000   0.0000000
## Hhold.fctrSKy:.clusterid.fctr4             0.0000000   0.0000000

#mdl_id <-"RFE.X.glmnet"; glb_featsimp_df <- myget_feats_importance(glb_models_lst[[mdl_id]], glb_featsimp_df); glb_featsimp_df[, paste0(mdl_id, ".imp")] <- glb_featsimp_df$imp; print(glb_featsimp_df)
#print(head(sbst_featsimp_df <- subset(glb_featsimp_df, is.na(RFE.X.glmnet.imp) | (abs(RFE.X.YeoJohnson.glmnet.imp - RFE.X.glmnet.imp) > 0.0001), select=-imp)))
#print(orderBy(~ -cor.y.abs, subset(glb_feats_df, id %in% c(row.names(sbst_featsimp_df), "startprice.dcm1.is9", "D.weight.post.stop.sum"))))

# Used again in fit.data.training & predict.data.new chunks
glb_analytics_diag_plots <- function(obs_df, mdl_id, prob_threshold=NULL) {
    if (!is.null(featsimp_df <- glb_featsimp_df)) {
        featsimp_df$feat <- gsub("`(.*?)`", "\\1", row.names(featsimp_df))    
        featsimp_df$feat.interact <- gsub("(.*?):(.*)", "\\2", featsimp_df$feat)
        featsimp_df$feat <- gsub("(.*?):(.*)", "\\1", featsimp_df$feat)    
        featsimp_df$feat.interact <- 
            ifelse(featsimp_df$feat.interact == featsimp_df$feat, 
                                            NA, featsimp_df$feat.interact)
        featsimp_df$feat <- 
            gsub("(.*?)\\.fctr(.*)", "\\1\\.fctr", featsimp_df$feat)
        featsimp_df$feat.interact <- 
            gsub("(.*?)\\.fctr(.*)", "\\1\\.fctr", featsimp_df$feat.interact) 
        featsimp_df <- orderBy(~ -imp.max, 
            summaryBy(imp ~ feat + feat.interact, data=featsimp_df,
                      FUN=max))    
        #rex_str=":(.*)"; txt_vctr=tail(featsimp_df$feat); ret_lst <- regexec(rex_str, txt_vctr); ret_lst <- regmatches(txt_vctr, ret_lst); ret_vctr <- sapply(1:length(ret_lst), function(pos_ix) ifelse(length(ret_lst[[pos_ix]]) > 0, ret_lst[[pos_ix]], "")); print(ret_vctr <- ret_vctr[ret_vctr != ""])    
        
        featsimp_df <- subset(featsimp_df, !is.na(imp.max))
        if (nrow(featsimp_df) > 5) {
            warning("Limiting important feature scatter plots to 5 out of ",
                    nrow(featsimp_df))
            featsimp_df <- head(featsimp_df, 5)
        }
        
    #     if (!all(is.na(featsimp_df$feat.interact)))
    #         stop("not implemented yet")
        rsp_var_out <- mygetPredictIds(glb_rsp_var, mdl_id)$value
        for (var in featsimp_df$feat) {
            plot_df <- melt(obs_df, id.vars = var, 
                            measure.vars = c(glb_rsp_var, rsp_var_out))
    
            print(myplot_scatter(plot_df, var, "value", colorcol_name = "variable",
                                facet_colcol_name = "variable", jitter = TRUE) + 
                          guides(color = FALSE))
        }
    }
    
    if (glb_is_regression) {
        if (is.null(featsimp_df) || (nrow(featsimp_df) == 0))
            warning("No important features in glb_fin_mdl") else
            print(myplot_prediction_regression(df=obs_df, 
                        feat_x=ifelse(nrow(featsimp_df) > 1, featsimp_df$feat[2],
                                      ".rownames"), 
                                               feat_y=featsimp_df$feat[1],
                        rsp_var=glb_rsp_var, rsp_var_out=rsp_var_out,
                        id_vars=glbFeatsId)
    #               + facet_wrap(reformulate(featsimp_df$feat[2])) # if [1 or 2] is a factor
    #               + geom_point(aes_string(color="<col_name>.fctr")) #  to color the plot
                  )
    }    
    
    if (glb_is_classification) {
        require(lazyeval)
        
        if (is.null(featsimp_df) || (nrow(featsimp_df) == 0))
            warning("No features in selected model are statistically important")
        else print(myplot_prediction_classification(df = obs_df, 
                                feat_x = ifelse(nrow(featsimp_df) > 1, 
                                                featsimp_df$feat[2], ".rownames"),
                                               feat_y = featsimp_df$feat[1],
                                                rsp_var = glb_rsp_var, 
                                                rsp_var_out = rsp_var_out, 
                                                id_vars = glbFeatsId,
                                                prob_threshold = prob_threshold))
    }    
}

if (glb_is_classification && glb_is_binomial)
    glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id = glbMdlSelId, 
            prob_threshold = glb_models_df[glb_models_df$id == glbMdlSelId, 
                                           "opt.prob.threshold.OOB"]) else
    glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id = glbMdlSelId)

## [1] "Min/Max Boundaries: "
## [1] "Inaccurate: "
##   USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 1       1          D                          0.469114
## 2      18          D                          0.469114
## 3      54          D                          0.469114
## 4      82          D                          0.469114
## 5     103          D                          0.469114
## 6     221          D                          0.469114
##   Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 1                            R                             TRUE
## 2                            R                             TRUE
## 3                            R                             TRUE
## 4                            R                             TRUE
## 5                            R                             TRUE
## 6                            R                             TRUE
##   Party.fctr.All.X..rcv.glmnet.err.abs Party.fctr.All.X..rcv.glmnet.is.acc
## 1                             0.530886                               FALSE
## 2                             0.530886                               FALSE
## 3                             0.530886                               FALSE
## 4                             0.530886                               FALSE
## 5                             0.530886                               FALSE
## 6                             0.530886                               FALSE
##   Party.fctr.All.X..rcv.glmnet.accurate Party.fctr.All.X..rcv.glmnet.error
## 1                                 FALSE                          -0.280886
## 2                                 FALSE                          -0.280886
## 3                                 FALSE                          -0.280886
## 4                                 FALSE                          -0.280886
## 5                                 FALSE                          -0.280886
## 6                                 FALSE                          -0.280886
##     USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 8       361          D                         0.4691140
## 11      460          D                         0.4691140
## 40     1759          D                         0.4691140
## 373    2655          D                         0.5232774
## 403    1321          D                         0.5286893
## 593     497          D                         0.7062498
##     Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 8                              R                             TRUE
## 11                             R                             TRUE
## 40                             R                             TRUE
## 373                            R                             TRUE
## 403                            R                             TRUE
## 593                            R                             TRUE
##     Party.fctr.All.X..rcv.glmnet.err.abs
## 8                              0.5308860
## 11                             0.5308860
## 40                             0.5308860
## 373                            0.4767226
## 403                            0.4713107
## 593                            0.2937502
##     Party.fctr.All.X..rcv.glmnet.is.acc
## 8                                 FALSE
## 11                                FALSE
## 40                                FALSE
## 373                               FALSE
## 403                               FALSE
## 593                               FALSE
##     Party.fctr.All.X..rcv.glmnet.accurate
## 8                                   FALSE
## 11                                  FALSE
## 40                                  FALSE
## 373                                 FALSE
## 403                                 FALSE
## 593                                 FALSE
##     Party.fctr.All.X..rcv.glmnet.error
## 8                          -0.28088603
## 11                         -0.28088603
## 40                         -0.28088603
## 373                        -0.22672265
## 403                        -0.22131067
## 593                        -0.04375019
##     USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 589    4662          D                         0.6580649
## 590    5410          D                         0.6580649
## 591    6477          D                         0.6580649
## 592    6885          D                         0.6580649
## 593     497          D                         0.7062498
## 594    1007          D                         0.7062498
##     Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 589                            R                             TRUE
## 590                            R                             TRUE
## 591                            R                             TRUE
## 592                            R                             TRUE
## 593                            R                             TRUE
## 594                            R                             TRUE
##     Party.fctr.All.X..rcv.glmnet.err.abs
## 589                            0.3419351
## 590                            0.3419351
## 591                            0.3419351
## 592                            0.3419351
## 593                            0.2937502
## 594                            0.2937502
##     Party.fctr.All.X..rcv.glmnet.is.acc
## 589                               FALSE
## 590                               FALSE
## 591                               FALSE
## 592                               FALSE
## 593                               FALSE
## 594                               FALSE
##     Party.fctr.All.X..rcv.glmnet.accurate
## 589                                 FALSE
## 590                                 FALSE
## 591                                 FALSE
## 592                                 FALSE
## 593                                 FALSE
## 594                                 FALSE
##     Party.fctr.All.X..rcv.glmnet.error
## 589                        -0.09193505
## 590                        -0.09193505
## 591                        -0.09193505
## 592                        -0.09193505
## 593                        -0.04375019
## 594                        -0.04375019

if (!is.null(glbFeatsCategory)) {
    glbLvlCategory <- merge(glbLvlCategory, 
            myget_category_stats(obs_df = glbObsFit, mdl_id = glbMdlSelId, 
                                 label = "fit"), 
                            by = glbFeatsCategory, all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    glbLvlCategory <- merge(glbLvlCategory, 
            myget_category_stats(obs_df = glbObsOOB, mdl_id = glbMdlSelId,
                                 label="OOB"),
                          #by=glbFeatsCategory, all=TRUE) glb_ctgry-df already contains .n.OOB ?
                          all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    if (any(grepl("OOB", glbMdlMetricsEval)))
        print(orderBy(~-err.abs.OOB.mean, glbLvlCategory)) else
            print(orderBy(~-err.abs.fit.mean, glbLvlCategory))
    print(colSums(glbLvlCategory[, -grep(glbFeatsCategory, names(glbLvlCategory))]))
}

##     Hhold.fctr .n.OOB .n.Fit .n.Tst .freqRatio.Fit .freqRatio.OOB
## N            N     83    367    102     0.08250899    0.074107143
## SKy        SKy     53    147     65     0.03304856    0.047321429
## MKn        MKn    136    516    169     0.11600719    0.121428571
## SKn        SKn    511   1920    638     0.43165468    0.456250000
## MKy        MKy    298   1296    371     0.29136691    0.266071429
## PKn        PKn     30    150     37     0.03372302    0.026785714
## PKy        PKy      9     52     10     0.01169065    0.008035714
##     .freqRatio.Tst err.abs.fit.sum err.abs.fit.mean .n.fit err.abs.OOB.sum
## N      0.073275862       183.38970        0.4996995    367        41.38970
## SKy    0.046695402        72.49587        0.4931692    147        26.41393
## MKn    0.121408046       255.65579        0.4954570    516        67.71214
## SKn    0.458333333       938.03522        0.4885600   1920       251.54176
## MKy    0.266522989       638.59067        0.4927397   1296       146.66848
## PKn    0.026580460        60.57196        0.4038131    150        14.73205
## PKy    0.007183908        24.83285        0.4775549     52         4.38788
##     err.abs.OOB.mean
## N          0.4986711
## SKy        0.4983761
## MKn        0.4978833
## SKn        0.4922539
## MKy        0.4921761
## PKn        0.4910685
## PKy        0.4875422
##           .n.OOB           .n.Fit           .n.Tst   .freqRatio.Fit 
##      1120.000000      4448.000000      1392.000000         1.000000 
##   .freqRatio.OOB   .freqRatio.Tst  err.abs.fit.sum err.abs.fit.mean 
##         1.000000         1.000000      2173.572078         3.350993 
##           .n.fit  err.abs.OOB.sum err.abs.OOB.mean 
##      4448.000000       552.845950         3.457971

write.csv(glbObsOOB[, c(glbFeatsId, 
                grep(glb_rsp_var, names(glbObsOOB), fixed=TRUE, value=TRUE))], 
    paste0(gsub(".", "_", paste0(glbOut$pfx, glbMdlSelId), fixed=TRUE), 
           "_OOBobs.csv"), row.names=FALSE)

fit.models_2_chunk_df <- 
    myadd_chunk(NULL, "fit.models_2_bgn", label.minor = "teardown")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_2_bgn          1          0    teardown 323.555  NA      NA

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc=FALSE)

##        label step_major step_minor label_minor     bgn     end elapsed
## 6 fit.models          4          2           2 312.890 323.566  10.676
## 7 fit.models          4          3           3 323.566      NA      NA

# if (sum(is.na(glbObsAll$D.P.http)) > 0)
#         stop("fit.models_3: Why is this happening ?")

#stop(here"); glb2Sav()
sync_glb_obs_df <- function() {
    # Merge or cbind ?
    for (col in setdiff(names(glbObsFit), names(glbObsTrn)))
        glbObsTrn[glbObsTrn$.lcn == "Fit", col] <<- glbObsFit[, col]
    for (col in setdiff(names(glbObsFit), names(glbObsAll)))
        glbObsAll[glbObsAll$.lcn == "Fit", col] <<- glbObsFit[, col]
    if (all(is.na(glbObsNew[, glb_rsp_var])))
        for (col in setdiff(names(glbObsOOB), names(glbObsTrn)))
            glbObsTrn[glbObsTrn$.lcn == "OOB", col] <<- glbObsOOB[, col]
    for (col in setdiff(names(glbObsOOB), names(glbObsAll)))
        glbObsAll[glbObsAll$.lcn == "OOB", col] <<- glbObsOOB[, col]
}
sync_glb_obs_df()
    
print(setdiff(names(glbObsNew), names(glbObsAll)))

## character(0)

replay.petrisim(pn = glb_analytics_pn, 
    replay.trans = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "model.selected")), flip_coord = TRUE)

## time trans    "bgn " "fit.data.training.all " "predict.data.new " "end " 
## 0.0000   multiple enabled transitions:  data.training.all data.new model.selected    firing:  model.selected 
## 1.0000    3   2 1 0 0

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.data.training", major.inc = TRUE)

##               label step_major step_minor label_minor     bgn     end
## 7        fit.models          4          3           3 323.566 327.504
## 8 fit.data.training          5          0           0 327.504      NA
##   elapsed
## 7   3.938
## 8      NA

Step `5.0: fit data training`

#load(paste0(glb_inp_pfx, "dsk.RData"))

if (!is.null(glbMdlFinId) && (glbMdlFinId %in% names(glb_models_lst))) {
    warning("Final model same as user selected model")
    glb_fin_mdl <- glb_models_lst[[glbMdlFinId]]
} else 
# if (nrow(glbObsFit) + length(glbObsFitOutliers) == nrow(glbObsTrn))
if (!all(is.na(glbObsNew[, glb_rsp_var])))
{    
    warning("Final model same as glbMdlSelId")
    glbMdlFinId <- paste0("Final.", glbMdlSelId)
    glb_fin_mdl <- glb_sel_mdl
    glb_models_lst[[glbMdlFinId]] <- glb_fin_mdl
    mdlDf <- glb_models_df[glb_models_df$id == glbMdlSelId, ]
    mdlDf$id <- glbMdlFinId
    glb_models_df <- rbind(glb_models_df, mdlDf)
} else {    
            if (grepl("RFE\\.X", names(glbMdlFamilies))) {
                indepVar <- mygetIndepVar(glb_feats_df)
                rfe_trn_results <- 
                    myrun_rfe(glbObsTrn, indepVar, glbRFESizes[["Final"]])
                if (!isTRUE(all.equal(sort(predictors(rfe_trn_results)),
                                      sort(predictors(rfe_fit_results))))) {
                    print("Diffs predictors(rfe_trn_results) vs. predictors(rfe_fit_results):")
                    print(setdiff(predictors(rfe_trn_results), predictors(rfe_fit_results)))
                    print("Diffs predictors(rfe_fit_results) vs. predictors(rfe_trn_results):")
                    print(setdiff(predictors(rfe_fit_results), predictors(rfe_trn_results)))
            }
        }
    # }    

    if (grepl("Ensemble", glbMdlSelId)) {
        # Find which models are relevant
        mdlimp_df <- subset(myget_feats_importance(glb_sel_mdl), imp > 5)
        # Fit selected models on glbObsTrn
        for (mdl_id in gsub(".prob", "", 
gsub(mygetPredictIds(glb_rsp_var)$value, "", row.names(mdlimp_df), fixed = TRUE),
                            fixed = TRUE)) {
            mdl_id_components <- unlist(strsplit(mdl_id, "[.]"))
            mdlIdPfx <- paste0(c(head(mdl_id_components, -1), "Train"), 
                               collapse = ".")
            if (grepl("RFE\\.X\\.", mdlIdPfx)) 
                mdlIndepVars <- myadjustInteractionFeats(glb_feats_df, myextract_actual_feats(
                    predictors(rfe_trn_results))) else
                mdlIndepVars <- trim(unlist(
            strsplit(glb_models_df[glb_models_df$id == mdl_id, "feats"], "[,]")))
            ret_lst <- 
                myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                        id.prefix = mdlIdPfx, 
                        type = glb_model_type, tune.df = glbMdlTuneParams,
                        trainControl.method = "repeatedcv",
                        trainControl.number = glb_rcv_n_folds,
                        trainControl.repeats = glb_rcv_n_repeats,
                        trainControl.classProbs = glb_is_classification,
                        trainControl.summaryFunction = glbMdlMetricSummaryFn,
                        train.metric = glbMdlMetricSummary, 
                        train.maximize = glbMdlMetricMaximize,    
                        train.method = tail(mdl_id_components, 1))),
                    indepVar = mdlIndepVars,
                    rsp_var = glb_rsp_var, 
                    fit_df = glbObsTrn, OOB_df = NULL)
            
            glbObsTrn <- glb_get_predictions(df = glbObsTrn,
                                                mdl_id = tail(glb_models_df$id, 1), 
                                                rsp_var = glb_rsp_var,
                                                prob_threshold_def = 
                    subset(glb_models_df, id == mdl_id)$opt.prob.threshold.OOB)
            glbObsNew <- glb_get_predictions(df = glbObsNew,
                                                mdl_id = tail(glb_models_df$id, 1), 
                                                rsp_var = glb_rsp_var,
                                                prob_threshold_def = 
                    subset(glb_models_df, id == mdl_id)$opt.prob.threshold.OOB)
        }    
    }
    
    # "Final" model
    if ((model_method <- glb_sel_mdl$method) == "custom")
        # get actual method from the mdl_id
        model_method <- tail(unlist(strsplit(glbMdlSelId, "[.]")), 1)
        
    if (grepl("Ensemble", glbMdlSelId)) {
        # Find which models are relevant
        mdlimp_df <- subset(myget_feats_importance(glb_sel_mdl), imp > 5)
        if (glb_is_classification && glb_is_binomial)
            indepVar <- gsub("(.*)\\.(.*)\\.prob", "\\1\\.Train\\.\\2\\.prob",
                                    row.names(mdlimp_df)) else
            indepVar <- gsub("(.*)\\.(.*)", "\\1\\.Train\\.\\2",
                                    row.names(mdlimp_df))
    } else 
    if (grepl("RFE.X", glbMdlSelId, fixed = TRUE)) {
        indepVar <- myextract_actual_feats(predictors(rfe_trn_results))
    } else indepVar <- 
                trim(unlist(strsplit(glb_models_df[glb_models_df$id ==
                                                   glbMdlSelId
                                                   , "feats"], "[,]")))
        
    if (!is.null(glb_preproc_methods) &&
        ((match_pos <- regexpr(gsub(".", "\\.", 
                                    paste(glb_preproc_methods, collapse = "|"),
                                   fixed = TRUE), glbMdlSelId)) != -1))
        ths_preProcess <- str_sub(glbMdlSelId, match_pos, 
                                match_pos + attr(match_pos, "match.length") - 1) else
        ths_preProcess <- NULL                                      

    mdl_id_pfx <- ifelse(grepl("Ensemble", glbMdlSelId),
                                   "Final.Ensemble", "Final")
    
    trnobs_df <- glbObsTrn 
    if (!is.null(glbObsTrnOutliers[[mdl_id_pfx]])) {
        trnobs_df <- glbObsTrn[!(glbObsTrn[, glbFeatsId] %in% glbObsTrnOutliers[[mdl_id_pfx]]), ]
        print(sprintf("Outliers removed: %d", nrow(glbObsTrn) - nrow(trnobs_df)))
        print(setdiff(glbObsTrn[, glbFeatsId], trnobs_df[, glbFeatsId]))
    }    
        
    # Force fitting of Final.glm to identify outliers
    method_vctr <- unique(c(myparseMdlId(glbMdlSelId)$alg, glbMdlFamilies[["Final"]]))
    for (method in method_vctr) {
        #source("caret_nominalTrainWorkflow.R")
        
        # glmnet requires at least 2 indep vars
        if ((length(indepVar) == 1) && (method %in% "glmnet"))
            next
        
        ret_lst <- 
            myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                    id.prefix = mdl_id_pfx, 
                    type = glb_model_type, trainControl.method = "repeatedcv",
                    trainControl.number = glb_rcv_n_folds, 
                    trainControl.repeats = glb_rcv_n_repeats,
                    trainControl.classProbs = glb_is_classification,
                    trainControl.summaryFunction = glbMdlMetricSummaryFn,
                    trainControl.allowParallel = glbMdlAllowParallel,
                    train.metric = glbMdlMetricSummary, 
                    train.maximize = glbMdlMetricMaximize,    
                    train.method = method,
                    train.preProcess = ths_preProcess)),
                indepVar = indepVar, rsp_var = glb_rsp_var, 
                fit_df = trnobs_df, OOB_df = NULL)
        
        if ((length(method_vctr) == 1) || (method != "glm")) {
            glb_fin_mdl <- glb_models_lst[[length(glb_models_lst)]] 
            glbMdlFinId <- glb_models_df[length(glb_models_lst), "id"]
        }
    }
        
}

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Final##rcv#glmnet"
## [1] "    indepVar: Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.709000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 0.1, lambda = 0.000925 on full training set
## [1] "myfit_mdl: train complete: 4.478000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = mdl_id_pfx, : model's bestTune found at an extreme of
## tuneGrid for parameter: alpha

##             Length Class      Mode     
## a0            58   -none-     numeric  
## beta        1624   dgCMatrix  S4       
## df            58   -none-     numeric  
## dim            2   -none-     numeric  
## lambda        58   -none-     numeric  
## dev.ratio     58   -none-     numeric  
## nulldev        1   -none-     numeric  
## npasses        1   -none-     numeric  
## jerr           1   -none-     numeric  
## offset         1   -none-     logical  
## classnames     2   -none-     character
## call           5   -none-     call     
## nobs           1   -none-     numeric  
## lambdaOpt      1   -none-     numeric  
## xNames        28   -none-     character
## problemType    1   -none-     character
## tuneValue      2   data.frame list     
## obsLevels      2   -none-     character
## [1] "min lambda > lambdaOpt:"
##                    (Intercept)                         .rnorm 
##                     0.05674217                    -0.01683326 
##                  Hhold.fctrMKn                  Hhold.fctrMKy 
##                    -0.09892081                    -0.39327027 
##                  Hhold.fctrPKn                  Hhold.fctrPKy 
##                     0.85896305                     0.02841362 
##                  Hhold.fctrSKn                  Hhold.fctrSKy 
##                    -0.15728881                     0.56548214 
##   Hhold.fctrN:.clusterid.fctr2 Hhold.fctrMKn:.clusterid.fctr2 
##                    -0.06488410                     0.20661936 
## Hhold.fctrMKy:.clusterid.fctr2 Hhold.fctrPKn:.clusterid.fctr2 
##                     0.34547637                     0.58504330 
## Hhold.fctrPKy:.clusterid.fctr2 Hhold.fctrSKn:.clusterid.fctr2 
##                    -0.28751286                     0.70656256 
## Hhold.fctrSKy:.clusterid.fctr2   Hhold.fctrN:.clusterid.fctr3 
##                    -0.81611026                    -0.05028136 
## Hhold.fctrMKn:.clusterid.fctr3 Hhold.fctrMKy:.clusterid.fctr3 
##                     0.80626250                     1.02572623 
## Hhold.fctrPKn:.clusterid.fctr3 Hhold.fctrPKy:.clusterid.fctr3 
##                    -0.61036216                    -0.40346940 
## Hhold.fctrSKn:.clusterid.fctr3 Hhold.fctrSKy:.clusterid.fctr3 
##                     0.27587858                    -0.53956389 
## Hhold.fctrPKy:.clusterid.fctr4 Hhold.fctrSKn:.clusterid.fctr4 
##                     2.35706143                     0.39303011 
## [1] "max lambda < lambdaOpt:"
## [1] "Feats mismatch between coefs_left & rght:"
##  [1] "(Intercept)"                    ".rnorm"                        
##  [3] "Hhold.fctrMKn"                  "Hhold.fctrMKy"                 
##  [5] "Hhold.fctrPKn"                  "Hhold.fctrPKy"                 
##  [7] "Hhold.fctrSKn"                  "Hhold.fctrSKy"                 
##  [9] "Hhold.fctrN:.clusterid.fctr2"   "Hhold.fctrMKn:.clusterid.fctr2"
## [11] "Hhold.fctrMKy:.clusterid.fctr2" "Hhold.fctrPKn:.clusterid.fctr2"
## [13] "Hhold.fctrPKy:.clusterid.fctr2" "Hhold.fctrSKn:.clusterid.fctr2"
## [15] "Hhold.fctrSKy:.clusterid.fctr2" "Hhold.fctrN:.clusterid.fctr3"  
## [17] "Hhold.fctrMKn:.clusterid.fctr3" "Hhold.fctrMKy:.clusterid.fctr3"
## [19] "Hhold.fctrPKn:.clusterid.fctr3" "Hhold.fctrPKy:.clusterid.fctr3"
## [21] "Hhold.fctrSKn:.clusterid.fctr3" "Hhold.fctrSKy:.clusterid.fctr3"
## [23] "Hhold.fctrN:.clusterid.fctr4"   "Hhold.fctrMKn:.clusterid.fctr4"
## [25] "Hhold.fctrMKy:.clusterid.fctr4" "Hhold.fctrPKn:.clusterid.fctr4"
## [27] "Hhold.fctrPKy:.clusterid.fctr4" "Hhold.fctrSKn:.clusterid.fctr4"
## [29] "Hhold.fctrSKy:.clusterid.fctr4"
## [1] "myfit_mdl: train diagnostics complete: 5.088000 secs"

##          Prediction
## Reference    R    D
##         R 2579   38
##         D 2823  128
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##     0.48617098     0.02721565     0.47296322     0.49939324     0.52999282 
## AccuracyPValue  McnemarPValue 
##     1.00000000     0.00000000 
## [1] "myfit_mdl: predict complete: 9.197000 secs"
##                  id                                        feats
## 1 Final##rcv#glmnet Hhold.fctr,.rnorm,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              20                      3.756                 0.165
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.5741715    0.5651509    0.5831921       0.3939275
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                    0.7       0.6432223        0.5631569
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.4729632             0.4993932     0.1234567
##   max.AccuracySD.fit max.KappaSD.fit
## 1        0.007710589        0.016188
## [1] "myfit_mdl: exit: 9.214000 secs"

rm(ret_lst)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.data.training", major.inc=FALSE)

##               label step_major step_minor label_minor     bgn     end
## 8 fit.data.training          5          0           0 327.504 337.342
## 9 fit.data.training          5          1           1 337.342      NA
##   elapsed
## 8   9.838
## 9      NA

#stop(here"); glb2Sav()
if (glb_is_classification && glb_is_binomial) 
    prob_threshold <- glb_models_df[glb_models_df$id == glbMdlSelId,
                                        "opt.prob.threshold.OOB"] else 
    prob_threshold <- NULL

if (grepl("Ensemble", glbMdlFinId)) {
    # Get predictions for each model in ensemble; Outliers that have been moved to OOB might not have been predicted yet
    mdlEnsembleComps <- unlist(str_split(subset(glb_models_df, 
                                                id == glbMdlFinId)$feats, ","))
    if (glb_is_classification && glb_is_binomial)
        mdlEnsembleComps <- gsub("\\.prob$", "", mdlEnsembleComps)
    mdlEnsembleComps <- gsub(paste0("^", 
                        gsub(".", "\\.", mygetPredictIds(glb_rsp_var)$value, fixed = TRUE)),
                             "", mdlEnsembleComps)
    for (mdl_id in mdlEnsembleComps) {
        glbObsTrn <- glb_get_predictions(df = glbObsTrn, mdl_id = mdl_id, 
                                            rsp_var = glb_rsp_var,
                                            prob_threshold_def = prob_threshold)
        glbObsNew <- glb_get_predictions(df = glbObsNew, mdl_id = mdl_id, 
                                            rsp_var = glb_rsp_var,
                                            prob_threshold_def = prob_threshold)
    }    
}
glbObsTrn <- glb_get_predictions(df = glbObsTrn, mdl_id = glbMdlFinId, 
                                     rsp_var = glb_rsp_var,
                                    prob_threshold_def = prob_threshold)

## Warning in glb_get_predictions(df = glbObsTrn, mdl_id = glbMdlFinId,
## rsp_var = glb_rsp_var, : Using default probability threshold: 0.75

glb_featsimp_df <- myget_feats_importance(mdl=glb_fin_mdl,
                                          featsimp_df=glb_featsimp_df)
#glb_featsimp_df[, paste0(glbMdlFinId, ".imp")] <- glb_featsimp_df$imp
print(glb_featsimp_df)

##                                All.X..rcv.glmnet.imp Final..rcv.glmnet.imp
## Hhold.fctrPKy:.clusterid.fctr4            64.9623500           100.0000000
## Hhold.fctrMKy:.clusterid.fctr3            70.6541605            43.5171616
## Hhold.fctrPKn                            100.0000000            36.4421157
## Hhold.fctrSKy:.clusterid.fctr2             0.0000000            34.6240557
## Hhold.fctrMKn:.clusterid.fctr3            39.7725789            34.2062574
## Hhold.fctrSKn:.clusterid.fctr2            67.4604337            29.9764167
## Hhold.fctrPKn:.clusterid.fctr3             0.0000000            25.8950467
## Hhold.fctrPKn:.clusterid.fctr2            39.2939854            24.8208763
## Hhold.fctrSKy                              4.6944480            23.9909801
## Hhold.fctrSKy:.clusterid.fctr3             0.0000000            22.8913802
## Hhold.fctrPKy:.clusterid.fctr3             0.0000000            17.1174749
## Hhold.fctrMKy                             37.4295156            16.6847698
## Hhold.fctrSKn:.clusterid.fctr4             0.8619652            16.6745807
## Hhold.fctrMKy:.clusterid.fctr2             0.0000000            14.6570798
## Hhold.fctrPKy:.clusterid.fctr2             0.0000000            12.1979366
## Hhold.fctrSKn:.clusterid.fctr3             0.0000000            11.7043442
## Hhold.fctrMKn:.clusterid.fctr2             0.0000000             8.7659727
## Hhold.fctrSKn                              0.0000000             6.6730893
## Hhold.fctrMKn                              0.0000000             4.1967855
## Hhold.fctrN:.clusterid.fctr2               0.0000000             2.7527540
## Hhold.fctrN:.clusterid.fctr3               0.0000000             2.1332222
## Hhold.fctrPKy                              0.0000000             1.2054678
## .rnorm                                     0.0000000             0.7141628
## Hhold.fctrMKn:.clusterid.fctr4             0.0000000             0.0000000
## Hhold.fctrMKy:.clusterid.fctr4             0.0000000             0.0000000
## Hhold.fctrN:.clusterid.fctr4               0.0000000             0.0000000
## Hhold.fctrPKn:.clusterid.fctr4             0.0000000             0.0000000
## Hhold.fctrSKy:.clusterid.fctr4             0.0000000             0.0000000
##                                        imp
## Hhold.fctrPKy:.clusterid.fctr4 100.0000000
## Hhold.fctrMKy:.clusterid.fctr3  43.5171616
## Hhold.fctrPKn                   36.4421157
## Hhold.fctrSKy:.clusterid.fctr2  34.6240557
## Hhold.fctrMKn:.clusterid.fctr3  34.2062574
## Hhold.fctrSKn:.clusterid.fctr2  29.9764167
## Hhold.fctrPKn:.clusterid.fctr3  25.8950467
## Hhold.fctrPKn:.clusterid.fctr2  24.8208763
## Hhold.fctrSKy                   23.9909801
## Hhold.fctrSKy:.clusterid.fctr3  22.8913802
## Hhold.fctrPKy:.clusterid.fctr3  17.1174749
## Hhold.fctrMKy                   16.6847698
## Hhold.fctrSKn:.clusterid.fctr4  16.6745807
## Hhold.fctrMKy:.clusterid.fctr2  14.6570798
## Hhold.fctrPKy:.clusterid.fctr2  12.1979366
## Hhold.fctrSKn:.clusterid.fctr3  11.7043442
## Hhold.fctrMKn:.clusterid.fctr2   8.7659727
## Hhold.fctrSKn                    6.6730893
## Hhold.fctrMKn                    4.1967855
## Hhold.fctrN:.clusterid.fctr2     2.7527540
## Hhold.fctrN:.clusterid.fctr3     2.1332222
## Hhold.fctrPKy                    1.2054678
## .rnorm                           0.7141628
## Hhold.fctrMKn:.clusterid.fctr4   0.0000000
## Hhold.fctrMKy:.clusterid.fctr4   0.0000000
## Hhold.fctrN:.clusterid.fctr4     0.0000000
## Hhold.fctrPKn:.clusterid.fctr4   0.0000000
## Hhold.fctrSKy:.clusterid.fctr4   0.0000000

if (glb_is_classification && glb_is_binomial)
    glb_analytics_diag_plots(obs_df=glbObsTrn, mdl_id=glbMdlFinId, 
            prob_threshold=glb_models_df[glb_models_df$id == glbMdlSelId, 
                                         "opt.prob.threshold.OOB"]) else
    glb_analytics_diag_plots(obs_df=glbObsTrn, mdl_id=glbMdlFinId)

## [1] "Min/Max Boundaries: "
## [1] "Inaccurate: "
##   USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 1    4314          D                          0.469114
## 2     662          D                          0.469114
## 3    3381          D                                NA
## 4    2311          D                                NA
## 5    3406          D                          0.469114
## 6    4373          D                          0.469114
##   Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 1                            R                             TRUE
## 2                            R                             TRUE
## 3                         <NA>                               NA
## 4                         <NA>                               NA
## 5                            R                             TRUE
## 6                            R                             TRUE
##   Party.fctr.All.X..rcv.glmnet.err.abs Party.fctr.All.X..rcv.glmnet.is.acc
## 1                             0.530886                               FALSE
## 2                             0.530886                               FALSE
## 3                                   NA                                  NA
## 4                                   NA                                  NA
## 5                             0.530886                               FALSE
## 6                             0.530886                               FALSE
##   Party.fctr.Final..rcv.glmnet.prob Party.fctr.Final..rcv.glmnet
## 1                         0.4042977                            R
## 2                         0.4045484                            R
## 3                         0.4054978                            R
## 4                         0.4059905                            R
## 5                         0.4060469                            R
## 6                         0.4068453                            R
##   Party.fctr.Final..rcv.glmnet.err Party.fctr.Final..rcv.glmnet.err.abs
## 1                             TRUE                            0.5957023
## 2                             TRUE                            0.5954516
## 3                             TRUE                            0.5945022
## 4                             TRUE                            0.5940095
## 5                             TRUE                            0.5939531
## 6                             TRUE                            0.5931547
##   Party.fctr.Final..rcv.glmnet.is.acc
## 1                               FALSE
## 2                               FALSE
## 3                               FALSE
## 4                               FALSE
## 5                               FALSE
## 6                               FALSE
##   Party.fctr.Final..rcv.glmnet.accurate Party.fctr.Final..rcv.glmnet.error
## 1                                 FALSE                         -0.3457023
## 2                                 FALSE                         -0.3454516
## 3                                 FALSE                         -0.3445022
## 4                                 FALSE                         -0.3440095
## 5                                 FALSE                         -0.3439531
## 6                                 FALSE                         -0.3431547
##      USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 12      3649          D                         0.4691140
## 196      476          D                         0.4691140
## 790     2908          D                         0.5220593
## 918     3204          D                         0.5220593
## 1593    6177          D                         0.5220593
## 2781    3610          D                         0.5777544
##      Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 12                              R                             TRUE
## 196                             R                             TRUE
## 790                             R                             TRUE
## 918                             R                             TRUE
## 1593                            R                             TRUE
## 2781                            R                             TRUE
##      Party.fctr.All.X..rcv.glmnet.err.abs
## 12                              0.5308860
## 196                             0.5308860
## 790                             0.4779407
## 918                             0.4779407
## 1593                            0.4779407
## 2781                            0.4222456
##      Party.fctr.All.X..rcv.glmnet.is.acc Party.fctr.Final..rcv.glmnet.prob
## 12                                 FALSE                         0.4082715
## 196                                FALSE                         0.4162178
## 790                                FALSE                         0.4772844
## 918                                FALSE                         0.4828065
## 1593                               FALSE                         0.5400926
## 2781                               FALSE                         0.6791571
##      Party.fctr.Final..rcv.glmnet Party.fctr.Final..rcv.glmnet.err
## 12                              R                             TRUE
## 196                             R                             TRUE
## 790                             R                             TRUE
## 918                             R                             TRUE
## 1593                            R                             TRUE
## 2781                            R                             TRUE
##      Party.fctr.Final..rcv.glmnet.err.abs
## 12                              0.5917285
## 196                             0.5837822
## 790                             0.5227156
## 918                             0.5171935
## 1593                            0.4599074
## 2781                            0.3208429
##      Party.fctr.Final..rcv.glmnet.is.acc
## 12                                 FALSE
## 196                                FALSE
## 790                                FALSE
## 918                                FALSE
## 1593                               FALSE
## 2781                               FALSE
##      Party.fctr.Final..rcv.glmnet.accurate
## 12                                   FALSE
## 196                                  FALSE
## 790                                  FALSE
## 918                                  FALSE
## 1593                                 FALSE
## 2781                                 FALSE
##      Party.fctr.Final..rcv.glmnet.error
## 12                          -0.34172854
## 196                         -0.33378216
## 790                         -0.27271564
## 918                         -0.26719348
## 1593                        -0.20990745
## 2781                        -0.07084288
##      USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 2898    3895          R                         0.7062498
## 2899    3288          R                         0.7062498
## 2900    2698          R                                NA
## 2901    1236          R                         0.7062498
## 2902    1610          R                         0.7062498
## 2903     626          R                         0.6121203
##      Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 2898                            R                            FALSE
## 2899                            R                            FALSE
## 2900                         <NA>                               NA
## 2901                            R                            FALSE
## 2902                            R                            FALSE
## 2903                            R                            FALSE
##      Party.fctr.All.X..rcv.glmnet.err.abs
## 2898                            0.7062498
## 2899                            0.7062498
## 2900                                   NA
## 2901                            0.7062498
## 2902                            0.7062498
## 2903                            0.6121203
##      Party.fctr.All.X..rcv.glmnet.is.acc Party.fctr.Final..rcv.glmnet.prob
## 2898                                TRUE                         0.8195896
## 2899                                TRUE                         0.8209868
## 2900                                  NA                         0.8210804
## 2901                                TRUE                         0.8212427
## 2902                                TRUE                         0.8230768
## 2903                                TRUE                         0.9190803
##      Party.fctr.Final..rcv.glmnet Party.fctr.Final..rcv.glmnet.err
## 2898                            D                             TRUE
## 2899                            D                             TRUE
## 2900                            D                             TRUE
## 2901                            D                             TRUE
## 2902                            D                             TRUE
## 2903                            D                             TRUE
##      Party.fctr.Final..rcv.glmnet.err.abs
## 2898                            0.8195896
## 2899                            0.8209868
## 2900                            0.8210804
## 2901                            0.8212427
## 2902                            0.8230768
## 2903                            0.9190803
##      Party.fctr.Final..rcv.glmnet.is.acc
## 2898                               FALSE
## 2899                               FALSE
## 2900                               FALSE
## 2901                               FALSE
## 2902                               FALSE
## 2903                               FALSE
##      Party.fctr.Final..rcv.glmnet.accurate
## 2898                                 FALSE
## 2899                                 FALSE
## 2900                                 FALSE
## 2901                                 FALSE
## 2902                                 FALSE
## 2903                                 FALSE
##      Party.fctr.Final..rcv.glmnet.error
## 2898                         0.06958955
## 2899                         0.07098684
## 2900                         0.07108042
## 2901                         0.07124266
## 2902                         0.07307676
## 2903                         0.16908028

dsp_feats_vctr <- c(NULL)
for(var in grep(".imp", names(glb_feats_df), fixed=TRUE, value=TRUE))
    dsp_feats_vctr <- union(dsp_feats_vctr, 
                            glb_feats_df[!is.na(glb_feats_df[, var]), "id"])

# print(glbObsTrn[glbObsTrn$UniqueID %in% FN_OOB_ids, 
#                     grep(glb_rsp_var, names(glbObsTrn), value=TRUE)])

print(setdiff(names(glbObsTrn), names(glbObsAll)))

## [1] "Party.fctr.Final..rcv.glmnet.prob"   
## [2] "Party.fctr.Final..rcv.glmnet"        
## [3] "Party.fctr.Final..rcv.glmnet.err"    
## [4] "Party.fctr.Final..rcv.glmnet.err.abs"
## [5] "Party.fctr.Final..rcv.glmnet.is.acc"

for (col in setdiff(names(glbObsTrn), names(glbObsAll)))
    # Merge or cbind ?
    glbObsAll[glbObsAll$.src == "Train", col] <- glbObsTrn[, col]

print(setdiff(names(glbObsFit), names(glbObsAll)))

## character(0)

print(setdiff(names(glbObsOOB), names(glbObsAll)))

## character(0)

for (col in setdiff(names(glbObsOOB), names(glbObsAll)))
    # Merge or cbind ?
    glbObsAll[glbObsAll$.lcn == "OOB", col] <- glbObsOOB[, col]
    
print(setdiff(names(glbObsNew), names(glbObsAll)))

## character(0)

#glb2Sav(); all.equal(savObsAll, glbObsAll); all.equal(sav_models_lst, glb_models_lst)
#load(file = paste0(glbOut$pfx, "dsk_knitr.RData"))
#cmpCols <- names(glbObsAll)[!grepl("\\.Final\\.", names(glbObsAll))]; all.equal(savObsAll[, cmpCols], glbObsAll[, cmpCols]); all.equal(savObsAll[, "H.P.http"], glbObsAll[, "H.P.http"]); 

replay.petrisim(pn = glb_analytics_pn, 
    replay.trans = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "data.training.all.prediction","model.final")), flip_coord = TRUE)

## time trans    "bgn " "fit.data.training.all " "predict.data.new " "end " 
## 0.0000   multiple enabled transitions:  data.training.all data.new model.selected    firing:  model.selected 
## 1.0000    3   2 1 0 0 
## 1.0000   multiple enabled transitions:  data.training.all data.new model.selected model.final data.training.all.prediction   firing:  data.training.all.prediction 
## 2.0000    5   2 0 0 1

## Warning in replay.petrisim(pn = glb_analytics_pn, replay.trans =
## (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, : Transition:
## model.final not enabled; adding missing token(s)

## Warning in replay.petrisim(pn = glb_analytics_pn, replay.trans
## = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, : Place:
## fit.data.training.all: added 1 missing token

## 2.0000   multiple enabled transitions:  data.training.all data.new model.selected model.final data.training.all.prediction   firing:  model.final 
## 3.0000    4   2 0 1 1

glb_chunks_df <- myadd_chunk(glb_chunks_df, "predict.data.new", major.inc = TRUE)

##                label step_major step_minor label_minor     bgn     end
## 9  fit.data.training          5          1           1 337.342 346.936
## 10  predict.data.new          6          0           0 346.937      NA
##    elapsed
## 9    9.594
## 10      NA

Step `6.0: predict data new`

## Warning in glb_get_predictions(obs_df, mdl_id = glbMdlFinId, rsp_var =
## glb_rsp_var, : Using default probability threshold: 0.75

## Warning in glb_get_predictions(obs_df, mdl_id = glbMdlFinId, rsp_var =
## glb_rsp_var, : Using default probability threshold: 0.75

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## NULL

## [1] "OOBobs total range outliers: 0"
## [1] "newobs total range outliers: 0"

## [1] 0.75

## [1] "glbMdlSelId: All.X##rcv#glmnet"

## [1] "glbMdlFinId: Final##rcv#glmnet"

## [1] "Cross Validation issues:"
##        MFO###myMFO_classfr  Random###myrandom_classfr 
##                          0                          0 
## Max.cor.Y.rcv.1X1###glmnet 
##                          0

##                            max.Accuracy.OOB max.AUCROCR.OOB
## All.X##rcv#glm                    0.4705357       0.4324359
## Random###myrandom_classfr         0.4696429       0.5191202
## MFO###myMFO_classfr               0.4696429       0.5000000
## Max.cor.Y.rcv.1X1###glmnet        0.4696429       0.4525163
## Max.cor.Y##rcv#rpart              0.4696429       0.4470465
## Low.cor.X##rcv#glmnet             0.4696429       0.4464208
## All.X##rcv#glmnet                 0.4696429       0.4464208
## Final##rcv#glmnet                        NA              NA
##                            max.AUCpROC.OOB max.Accuracy.fit
## All.X##rcv#glm                   0.5576807        0.5647478
## Random###myrandom_classfr        0.5235690        0.4700989
## MFO###myMFO_classfr              0.5000000        0.4700989
## Max.cor.Y.rcv.1X1###glmnet       0.5546626        0.4880845
## Max.cor.Y##rcv#rpart             0.5502714        0.5696192
## Low.cor.X##rcv#glmnet            0.5229705        0.5678975
## All.X##rcv#glmnet                0.5229705        0.5678975
## Final##rcv#glmnet                       NA        0.5631569
##                            opt.prob.threshold.fit opt.prob.threshold.OOB
## All.X##rcv#glm                               0.70                   0.90
## Random###myrandom_classfr                    0.55                   0.55
## MFO###myMFO_classfr                          0.50                   0.50
## Max.cor.Y.rcv.1X1###glmnet                   0.70                   0.85
## Max.cor.Y##rcv#rpart                         0.70                   0.80
## Low.cor.X##rcv#glmnet                        0.65                   0.75
## All.X##rcv#glmnet                            0.65                   0.75
## Final##rcv#glmnet                            0.70                     NA

## [1] "All.X##rcv#glmnet OOB confusion matrix & accuracy: "
##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0

##     err.abs.fit.sum err.abs.OOB.sum err.abs.trn.sum err.abs.new.sum
## N         183.38970        41.38970       224.93015              NA
## SKy        72.49587        26.41393        93.67543              NA
## MKn       255.65579        67.71214       319.23871              NA
## SKn       938.03522       251.54176      1176.17051              NA
## MKy       638.59067       146.66848       767.08090              NA
## PKn        60.57196        14.73205        69.30674              NA
## PKy        24.83285         4.38788        25.16142              NA
##     .freqRatio.Fit .freqRatio.OOB .freqRatio.Tst .n.Fit .n.New.D .n.New.R
## N       0.08250899    0.074107143    0.073275862    367       NA      102
## SKy     0.03304856    0.047321429    0.046695402    147       NA       65
## MKn     0.11600719    0.121428571    0.121408046    516       NA      169
## SKn     0.43165468    0.456250000    0.458333333   1920       NA      638
## MKy     0.29136691    0.266071429    0.266522989   1296       NA      371
## PKn     0.03372302    0.026785714    0.026580460    150       12       25
## PKy     0.01169065    0.008035714    0.007183908     52        1        9
##     .n.OOB .n.Trn.D .n.Trn.R .n.Tst .n.fit .n.new .n.trn err.abs.OOB.mean
## N       83      230      220    102    367    102    450        0.4986711
## SKy     53      119       81     65    147     65    200        0.4983761
## MKn    136      344      308    169    516    169    652        0.4978833
## SKn    511     1340     1091    638   1920    638   2431        0.4922539
## MKy    298      752      842    371   1296    371   1594        0.4921761
## PKn     30      131       49     37    150     37    180        0.4910685
## PKy      9       35       26     10     52     10     61        0.4875422
##     err.abs.fit.mean err.abs.new.mean err.abs.trn.mean
## N          0.4996995               NA        0.4998448
## SKy        0.4931692               NA        0.4683772
## MKn        0.4954570               NA        0.4896299
## SKn        0.4885600               NA        0.4838217
## MKy        0.4927397               NA        0.4812302
## PKn        0.4038131               NA        0.3850375
## PKy        0.4775549               NA        0.4124823
##  err.abs.fit.sum  err.abs.OOB.sum  err.abs.trn.sum  err.abs.new.sum 
##      2173.572078       552.845950      2675.563871               NA 
##   .freqRatio.Fit   .freqRatio.OOB   .freqRatio.Tst           .n.Fit 
##         1.000000         1.000000         1.000000      4448.000000 
##         .n.New.D         .n.New.R           .n.OOB         .n.Trn.D 
##               NA      1379.000000      1120.000000      2951.000000 
##         .n.Trn.R           .n.Tst           .n.fit           .n.new 
##      2617.000000      1392.000000      4448.000000      1392.000000 
##           .n.trn err.abs.OOB.mean err.abs.fit.mean err.abs.new.mean 
##      5568.000000         3.457971         3.350993               NA 
## err.abs.trn.mean 
##         3.220424

## [1] "Features Importance for selected models:"

##                                All.X..rcv.glmnet.imp Final..rcv.glmnet.imp
## Hhold.fctrPKn                            100.0000000              36.44212
## Hhold.fctrMKy:.clusterid.fctr3            70.6541605              43.51716
## Hhold.fctrSKn:.clusterid.fctr2            67.4604337              29.97642
## Hhold.fctrPKy:.clusterid.fctr4            64.9623500             100.00000
## Hhold.fctrMKn:.clusterid.fctr3            39.7725789              34.20626
## Hhold.fctrPKn:.clusterid.fctr2            39.2939854              24.82088
## Hhold.fctrMKy                             37.4295156              16.68477
## Hhold.fctrSKy                              4.6944480              23.99098
## Hhold.fctrSKn:.clusterid.fctr4             0.8619652              16.67458
## Hhold.fctrSKy:.clusterid.fctr2             0.0000000              34.62406
## Hhold.fctrPKn:.clusterid.fctr3             0.0000000              25.89505
## Hhold.fctrSKy:.clusterid.fctr3             0.0000000              22.89138
## Hhold.fctrPKy:.clusterid.fctr3             0.0000000              17.11747
## Hhold.fctrMKy:.clusterid.fctr2             0.0000000              14.65708
## Hhold.fctrPKy:.clusterid.fctr2             0.0000000              12.19794
## Hhold.fctrSKn:.clusterid.fctr3             0.0000000              11.70434

## [1] "glbObsNew prediction stats:"

## 
##    R    D 
## 1379   13

##                   label step_major step_minor label_minor     bgn     end
## 10     predict.data.new          6          0           0 346.937 360.568
## 11 display.session.info          7          0           0 360.568      NA
##    elapsed
## 10  13.631
## 11      NA

Null Hypothesis ($\sf{H_{0}}$): mpg is not impacted by am_fctr.
The variance by am_fctr appears to be independent. #{r q1, cache=FALSE} # print(t.test(subset(cars_df, am_fctr == "automatic")$mpg, # subset(cars_df, am_fctr == "manual")$mpg, # var.equal=FALSE)$conf) # We reject the null hypothesis i.e. we have evidence to conclude that am_fctr impacts mpg (95% confidence). Manual transmission is better for miles per gallon versus automatic transmission.

##                      label step_major step_minor label_minor     bgn
## 1             cluster.data          1          0           0   6.109
## 2  partition.data.training          2          0           0 129.165
## 4               fit.models          4          0           0 245.736
## 5               fit.models          4          1           1 282.174
## 10        predict.data.new          6          0           0 346.937
## 6               fit.models          4          2           2 312.890
## 8        fit.data.training          5          0           0 327.504
## 9        fit.data.training          5          1           1 337.342
## 7               fit.models          4          3           3 323.566
## 3          select.features          3          0           0 242.864
##        end elapsed duration
## 1  129.164 123.055  123.055
## 2  242.864 113.699  113.699
## 4  282.173  36.437   36.437
## 5  312.889  30.716   30.715
## 10 360.568  13.631   13.631
## 6  323.566  10.676   10.676
## 8  337.342   9.838    9.838
## 9  346.936   9.594    9.594
## 7  327.504   3.938    3.938
## 3  245.735   2.871    2.871
## [1] "Total Elapsed Time: 360.568 secs"

Show of Hands:Votes: Party classification:: Votes_Q_02_cluster

bdanalytics

Introduction:

Synopsis:

Prediction Accuracy Enhancement Options:

[](.png)

Potential next steps include:

Analysis:

Step 1.0: cluster data

chunk option: eval=

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

```{r extract.features.image, cache=FALSE, echo=FALSE, fig.height=5, fig.width=5, eval=myevlChunk(glbChunks, glbOut$pfx)}

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx)} #{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx, keep = c(“glbFeatsCategory”,“glb_dsp_cols”))}

Step 2.0: partition data training

Step 3.0: select features

Step 4.0: fit models

Step 5.0: fit data training

Step 6.0: predict data new

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx)} #`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx, keep = c(“glbFeatsCategory”,“glb_dsp_cols”))}

Step `2.0: partition data training`

Step `3.0: select features`

Step `4.0: fit models`

Step `5.0: fit data training`

Step `6.0: predict data new`